Tech News - Data

Apache Hadoop 2.9.0 has been released. This is the first release of Hadoop 2.9 line, marking the start of Apache Hadoop 2.9.x series. It includes 30 new features with 500+ subtasks, in addition to 407 improvements and 790 bug fixes including new fixed issues since 2.8.2. For download instructions, users can refer to the Apache Hadoop release page. Apache Hadoop 2.9.0: Major features Here is a short overview of the major features and improvements that come with this release: Common HADOOP Resource Estimator HDFS HDFS Router based federation YARN YARN Timeline Service v.2 YARN Federation Opportunistic Containers YARN Web UI v.2 Changing queue configuration via API (supported only on the Capacity Scheduler) Update Resources and Execution Type of an allocated/running container (supported only on the Capacity Scheduler) More details on all the features, subtasks and bug fixes can be found in the change log and release notes. Getting Started The Hadoop documentation includes the information you need to get started using Hadoop. Begin with the Single Node Setup which shows you how to set up a single-node Hadoop installation. Then move on to the Cluster Setup to learn how to set up a multi-node Hadoop installation. A note from the team behind Apache Hadoop 2.9.0 "Although this release has been tested on fairly large clusters, production users can wait for a subsequent point release which will contain fixes from further stabilization and downstream adoption."

Amazon Web Services has announced the availability of two new versions of the AWS Deep Learning AMI: Conda-based AMI and Base AMI. The Conda-based AMI comes pre-installed with separate Python environments for deep learning frameworks created using Conda, while the Base AMI comes pre-installed with the foundational building blocks for deep learning. “Think of the Conda-based AMI as a fully baked virtual environment ready to run your deep learning code, for example, to train a neural network model. Think of the Base AMI as a clean slate to deploy your customized deep learning set up,” Amazon said in its announcement. Conda-based Deep Learning AMI The Conda-based AMI has Python environments for deep learning created using Conda—a popular open source package and environment management tool. In addition to the flexibility at the run-time environment, the AMI provides a visual interface that plugs straight into the Jupyter notebooks. “So you can switch in and out of environments, launch a notebook in an environment of your choice, and even reconfigure your environment—all with a single click, right from your Jupyter notebook browser. Our step-by-step guide walks you through these integrations and other Jupyter notebooks and tutorials,” Amazon said. The new Conda-based Deep Learning AMI comes packaged with the latest official releases of the following deep learning frameworks: Apache MXNet 0.12 with Gluon TensorFlow 1.4 Caffe2 0.8.1 PyTorch 0.2 CNTK 2.2 Theano 0.9 Keras 1.2.2 and Keras 2.0.9 The AMI also includes the following libraries and drivers for GPU acceleration on the cloud: CUDA 8 and 9 cuDNN 6 and 7 NCCL 2.0.5 libraries NVidia Driver 384.81 Deep Learning Base AMI The new Base AMI comes with GPU drivers and libraries to deploy your own customized deep learning models. If you are a developer who is contributing to open source deep learning framework enhancements or even creating a new deep learning engine, the Base AMI will provide the foundation to install your own custom configurations and code repositories to test out new framework features. By default, the AMI is configured with an NVidia CUDA 9 environment. However, we can switch to a CUDA 8 environment by reconfiguring the environment variable LD_LIBRARY_PATH. Simply replace the CUDA 9 portion of the environment variable string with its CUDA 8 equivalent. The Base AMI provides the following GPU drivers and libraries: CUDA 8 and 9 CuBLAS 8 and 9 CuDNN 6 and 7 glibc 2.18 OpenCV 3.2.0 NVIDIA driver 384.81 NCCL 2.0.5 Python 2 and 3 Amazon has set up new developer resources to help select the right AMI for your project. The AMI selection guide can be accessed here.

New tool Olympus, SAP's collaboration with Red Hat, blockchain-powered Visa B2B Connect, and Dialogflow Enterprise Edition in today's top stories around data science news. An instant REST API for any AI model Olympus – A new tool that instantly creates a REST API for any AI model Olympus is a command-line tool to deploy a pre-trained machine learning or deep learning model as a REST API, in seconds, thus eliminating the need for developers to manually create the REST APIs. “One of the key differences between Olympus and TensorFlow Serving is that, while TF Serving is optimized for the production environment, Olympus is currently more geared towards the development phase,” Olympus developers announced. To install Olympus, run the code pip install olympus. Big data meets containerization SAP Vora introduced into Red Hat OpenShift Container Platform SAP Vora solution on Red Hat OpenShift Container Platform is an integrated solution that pairs enterprise-grade Kubernetes with actionable big data insights. Key features of the integrated offering include: On-demand in-memory big data analytics, easier management of big data analytics at scale, easier integration of SAP Vora with SAP HANA, and better support for agile development around big data use cases. Visa B2B for cross-border corporate payments Visa kicks off pilot phase of “Visa B2B Connect” blockchain-based platform, commercial launch in mid-2018 Visa has announced the pilot phase of its blockchain-based platform Visa B2B Connect. The credit card giant had previewed the global payment platform in October 2016. Using blockchain-based architecture, Visa B2B Connect simplifies existing cross-border corporate payments by sending transactions over Visa’s network from the bank of origin directly to the recipient bank. Following the first phase, the commercial launch of the platform is planned for mid-2018. Google Dialogflow to power conversational interactions Google rolls out paid enterprise edition of Dialogflow with added speech integration Google has announced the beta release of enterprise edition of Dialogflow, its tool for building chatbots and other conversational applications. The enterprise edition offers greater flexibility and support for large-scale businesses, and has built-in support for speech recognition,  enabling developers to build rich voice-based applications. The enterprise Edition offers unlimited pay-as-you-go voice support. Companies like Uniqlo, Policybazaar.com and Strayer University have already used Dialogflow to design and deploy conversational experiences, Google said. Dialogflow was formerly known as API.AI, until its acquisition by Google last year.

Google has recently announced the enterprise edition of Dialogflow, its Chatbot API. Dialogflow is Google’s API for building chatbots as well as other conversational interfaces for mobile applications, websites, messaging platforms, and IoT devices. It uses machine learning and natural language processing in the backend to power it’s conversational interfaces. It also has a built-in speech recognition support and features new analytics capabilities. Now they have extended the API to the enterprises, allowing organizations to build these conversational apps for a large scale usage. According to Google, Dialogflow Enterprise Edition is a premium pay-as-you-go service. It is targeted at organizations in need of enterprise-grade services that can withstand changes based on user demands. As opposed to the small and medium business owners and individual developers for whom the standard version suffices. The enterprise edition also boasts of 24/7 support, SLAs, enterprise-level terms of service and complete data protection which is why companies are willing to pay a fee for adopting it. Here’s a quick overview of the differences between the standard and the enterprise version of Dialogflow: Source: https://cloud.google.com/dialogflow-enterprise/docs/editions Apart from this, the API is also a part of Google Cloud. So, it comes with the same support options as provided to cloud platform customers. The enterprise edition also supports unlimited text and voice interactions and higher usage quotas as compared to the free version. It's Enterprise Edition agent can be created using the Google Cloud Platform Console. Adding, editing or removing entities and intents to the agent can be done using console.dialogflow.com, or with the Dialogflow V2 API. Here’s a quick glance at some top features: Natural language Understanding, allows quick extraction and response of a user’s intent to implement natural and rich interactions between users and businesses. Over 30+ pre-built agents for quick and easy identification of custom entity types. An integrated code editor, to build native serverless applications linked with conversational interfaces through Cloud Functions for Firebase. Integration with Google Cloud Speech,  for voice interactions, support in a single API Cross-Platform and Multi-Language Agent, with 20+ languages supported over 14 different platforms. Uniqlo has used Dialogflow to create their shopping chatbot. Here are the views of Shinya Matsuyama, Director of Global digital commerce, Uniqlo: “Our shopping chatbot was developed using Dialogflow to offer a new type of shopping experience through a messaging interface, and responses are constantly being improved through machine learning. Going forward, we are also looking to expand the functionality to include voice recognition and multiple languages. ” According to the official documentation, the project is still in beta stage. Hence, it is not intended for real-time usage in critical applications. You can learn more about the project along with Quickstarts, How-to guides, and Tutorials here.

Continuing with last week’s trend, partnerships blossom among tech giants and open source projects, interactive computing and quantum computing are in vogue, and clouds get serious about AI. Here is a quick rundown of news in the data science space worth your attention!   News Highlights Microsoft showcases its edgy AI toolkit at Connect(); 2017 3 ways JupyterLab will revolutionize Interactive Computing What we are learning from Microsoft Connect(); 2017 Microsoft releases first test build of Windows Server 1803 Introducing Azure Databricks: Spark analytics comes to the cloud Amazon ups its A.I. game by launching Ironman to the cloud “We will raise a toast to Python 2” — NumPy announces transition plan to Python 3 Tensorflow Lite developer preview is Here Introducing Tile: A new machine learning language with auto-generating GPU Kernels Has IBM edged past Google in the battle for Quantum Supremacy? In other News 17th Nov.’ 17 – Headlines AWS unveils two new deep learning AMIs for machine learning practitioners Google’s bot analytics platform “Chatbase” now open to everyone Google’s BigQuery data transfer service is now generally available 16th Nov.’ 17 – Headlines Visual Basic Upgrade Companion 8.0 comes with additional machine learning Datameer introduces cloud-architected big data platform through AWS Hitachi announces new 100x faster technology for open source software based big data analytics Bitcoin Gold Launched 15th Nov.’ 17 – Headlines Google adds Multi-Region support in Cloud Spanner Twitter announces premium APIs, starts with Tweet search at $149/month Elasticsearch 6.0 released: sequence IDs, circuit breakers, index sorting key improvements IBM unveils Deep Learning Impact, updates Spectrum LSF Suites and Spectrum Conductor 14th Nov.’ 17 – Headlines Dell EMC announces high-performance computing bundles aimed at AI, deep learning Dell EMC announces new PowerEdge server designed specifically for HPC workloads Hewlett Packard announces set of upgraded HPC systems for AI Neural Fuzzing: Microsoft uses machine learning, deep neural networks for new vulnerability testing 13th Nov.’ 17 – Headlines Brytlyt announces visual analytics tool SpotLyt for billion row data sets Franz adds Triple Attribute security to AllegroGraph

At the ongoing Microsoft Connect(); 2017, Microsoft has unveiled their latest innovations in AI development platforms. The Connect(); conference this year is all about developing new tools and cloud services that help developers seize the growing opportunity around artificial intelligence and machine learning. Microsoft has made two major announcements to capture the AI market. Visual Studio Tools for AI Microsoft has announced new tools for its Visual Studio IDE specific for building AI applications. Visual Studio Tools for AI is currently in the beta stage and is an extension to the Visual Studio 2017. It allows developers, data scientists, and machine learning engineers to embed deep learning models into applications. They also have built-in support for popular machine learning frameworks such as Microsoft Cognitive Toolkit (CNTK), Google TensorFlow, Caffe2, and MXNet. It also comes packed with features such as custom metrics, history tracking, enterprise-ready collaboration, and data science reproducibility and auditing. Visual Studio Tools for AI allows interactive debugging of deep learning applications with built-in features like syntax highlighting, IntelliSense and text auto formatting. Training of AI models on the cloud is also possible using the integration with Azure Machine Learning. This integration also allows deploying a model into production. Visualization and monitoring of AI models is available using TensorBoard, which is an integrated open tool and can be run both locally and in remote VMs. Azure IoT Edge Microsoft sees IoT as a mission-critical business asset. With this in mind, they have developed a product for IoT solutions. Termed as Azure IoT Edge, it enables developers to run cloud intelligence on the edge of IoT devices. Azure IoT Edge can operate on Windows and Linux as well as on multiple hardware architectures (x64 and ARM). Developers can work on languages such as C#, C and Python to deploy models on Azure IoT Edge. The Azure IoT edge is a bundle of multiple components. With AI Toolkit, developers can start building AI applications. With Azure Machine learning, AI applications can be created, deployed, and managed with the toolkit on any framework. Azure Machine Learning also includes a set of pre-built AI models for common tasks. In addition, using the Azure IoT Hub, developers can deploy Edge modules on multiple IoT Edge devices. Using a combination of Azure Machine Learning, Azure Stream Analytics, Azure Functions, and any third-party code, a complex data pipeline can be created to build and test container-based workloads. This pipeline can be managed using the Azure IoT Hub. The customer reviews on Azure IoT edge have been positive up till now. Here’s what Matt Boujonnier, Analytics Application Architect at Schneider Electric says: "Azure IoT Edge provided an easy way to package and deploy our Machine Learning applications. Traditionally, machine learning is something that has only run in the cloud, but for many IoT scenarios that isn’t good enough, because you want to run your application as close as possible to any events. Now we have the flexibility to run it in the cloud or at the edge—wherever we need it to be." With the launch of these two new tools, Microsoft is catching up quickly with the likes of Google and IBM to capture the AI market and providing developers with an intelligent edge.

Amazon Web Services (AWS) has made building deep learning models easy for developers! Recently, AWS announced the availability of ONNX-MXNet, which is an open source Python package to import ONNX (Open Neural Network Exchange) deep learning models into Apache MXNet. Deep learning is a relatively new field and as such does not have multiple available methods for developers to build data models. Also, switching the deep learning models from their original framework that they are built on, to a new framework is difficult and time-consuming. This is one of the reasons why Microsoft and Facebook released ONNX in September this year. ONNX, an open source format for the deep learning models allows machines to learn tasks without the need of being programmed explicitly. Deep learning models trained on one framework are easily transferable to another with the help of the ONNX format, with no additional work. Apache MXNet is a fully featured and scalable deep learning framework. It offers APIs across popular languages such as Python, Scala, and R. It consists of a Gluon Interface which allows developers of different skill levels to begin with deep learning on the cloud, mobile applications, and also on edge devices. With a few Gluon code, developers are able to build different models such as, linear regression, convolutional networks and recurrent LSTMs (Long Short Term Memory). This further helps them to carry out tasks such as Object detection, recommendation, speech recognition, and even personalization. By providing ONNX format support for MXNet, developers can bring together the features of both ONNX and MXNet. This means, they can build and train models on various other deep learning frameworks including Microsoft Cognitive Toolkit, Caffe2, and so on. Developers can also import these models within MXNet to run them for inference using the MXNet engine, which is highly optimized and scalable. ONNX community will continue developing the ONNX format and the ecosystem. Facebook plans to add more interoperability that can expand ONNX for MXNet functionality. Plans for getting ONNX for MXNet core APIs is also on the cards. Amid all the good news, the deep learning community is still longing for one elusive partner. Google's entry into ONNX would be an icing on the cake.

SQL Operations Studio, Amazon's support for ONNX, and Google's Chatbase among today's trending stories in data science news. Announcing SQL Operations Studio for preview Microsoft announced that SQL Operations Studio is now available in preview. Users can install SQL Operations Studio (preview) as following: For Windows Download SQL Operations Studio (preview) for Windows. Browse to the downloaded file and extract it. Run sqlops-windowssqlops.exe For macOS Download SQL Operations Studio (preview) for macOS. To expand the contents of the zip, double-click it. To make SQL Operations Studio (preview) available in the Launchpad, drag sqlops.app to the Applications folder. For Linux Download SQL Operations Studio (preview) for Linux. To extract the file and launch SQL Operations Studio (preview), open a new Terminal window and type the following commands: cd ~ cp ~/Downloads/sqlops-linux-<version string>.tar.gz ~ tar -xvf ~/sqlops-linux-<version string>.tar.gz echo 'export PATH="$PATH:~/sqlops-linux-x64"' >> ~/.bashrc source ~/.bashrc sqlops Microsoft added a note that on Ubuntu and Redhat, users may have a missing dependency for libXScrnSaver. To install this dependency, following commands can be used: Ubuntu: sudo apt-get install libxss1 Redhat: yum install libXScrnSaver New announcements from Amazon Web services (AWS) AWS announces ONNX support for Apache MXNet The Open Neural Network Exchange (ONNX) deep-learning format, introduced in September by Microsoft and Facebook, has a new backer following Amazon Web Services’ decision to embrace the framework with a new open-source project. The AWS has released ONNX-MXNet, a method for allowing deep learning models built around the ONNX format to run on the Apache MXNet framework. MXNet is a fully featured and scalable deep learning framework that offers APIs across popular languages such as Python, Scala, and R. With ONNX format support for MXNet, developers can build and train models with other frameworks, such as PyTorch, Microsoft Cognitive Toolkit, or Caffe2, and import these models into MXNet to run them for inference using the MXNet highly optimized and scalable engine. AWS unveils two new deep learning AMIs for machine learning practitioners Amazon Web Services has announced the availability of two new versions of the AWS Deep Learning AMI: Conda-based AMI and Base AMI. The Conda-based AMI comes pre-installed with separate Python environments for deep learning frameworks created using Conda, while the Base AMI comes pre-installed with the foundational building blocks for deep learning. “Think of the Conda-based AMI as a fully baked virtual environment ready to run your deep learning code, for example, to train a neural network model. Think of the Base AMI as a clean slate to deploy your customized deep learning set up,” Amazon said in its release. The Conda-based AMI is packaged with latest official releases of the following deep learning frameworks: Apache MXNet 0.12 with Gluon, TensorFlow 1.4, Caffe2 0.8.1, PyTorch 0.2, CNTK 2.2, Theano 0.9, Keras 1.2.2, and Keras 2.0.9. The Base AMI comes with the CUDA 9 environment installed by default, but users can switch to a CUDA 8 environment. The Base AMI provides following GPU drivers and libraries: CUDA 8 and 9, CuBLAS 8 and 9, CuDNN 6 and 7, glibc 2.18, OpenCV 3.2.0, NVIDIA driver 384.81, NCCL 2.0.5, Python 2 and 3. Google's Chatbase and BigQuery Google’s bot analytics platform “Chatbase” now open to everyone More than six months after it quietly announced “Chatbase” during the I/O developer conference, Google has made the chatbot analytics platform open for public use. Chatbase helps developers analyze and optimize their bots better so that they can improve conversion rates and accuracy. Anyone can use Google’s Chatbase for free, similar to Google Analytics, and it’ll work across any platform, including Facebook Messenger, Kik, Slack, Viber, and Skype. Google’s BigQuery data transfer service is now generally available Google's BigQuery Data Transfer Service is now generally available, offering users a way to easily transfer data from supported SaaS applications in an automated fashion. So far, the service supports transfer from apps like AdWords, DoubleClick Campaign Manager, DoubleClick for Publishers, and YouTube Content and Channel Owner Reports. The service has some new features, including customer-managed scheduling, which lets customers set their own data delivery schedules. It also now offers a data delivery service-level agreement (SLA). Companies like Trivago and Zenith have already begun using the service, Google said. Pricing information can be found here.

Microsoft has released to Insiders the first test construct of its next Windows Server. Build 17035 of Windows Server is identical to its PC counterpart, codenamed "Redstone 4." Assuming Microsoft sticks to its six-month release cadence, both the client and server function updates are anticipated to be designated 1803 (for March 2018), and to start rolling out to mainstream users around April 2018. In Microsoft dictionary, Windows Server 1803 could, therefore, be the next "Semi-Annual Channel release" for Windows Server. As of this construct, Server Insiders get the selection of ISO layout or VHDX layout, with photographs pre-keyed, eliminating the need to input a key right through the setup. An important announcement in 17035 is the return of Storage Spaces Direct (S2D), which mysteriously disappeared from Windows Server 1709, allegedly because of quality issues. Since then, Microsoft has been promising “hyper-converged innovation” in “another release available very soon.” But now the tech leader has said that the software-defined storage tool is coming back with “some new and necessary updates” added to it including Data Deduplication and Resilient File System (ReFS). The only other addition to this construct is that builders can use localhost or loopback (127.0.0.1) to get admission to products and services running in containers on the host. With this new test build, Microsoft is making available Honolulu Technical Preview 1711 Construct 01003. Honolulu is a graphical control device for Windows Server. Inside Honolulu, Microsoft is making updates and adjustments to Remote Desktop, Windows 10 client management, Switch Embedded Teaming, and Data grid performance. In its announcement, Microsoft has listed the recognized problems with Server 17035, such as the cases when the base filtering engine (BFE) service may fail to start preventing the Windows Defender firewall (MpsSvc service) from starting, and that testing of the Windows core may fail because of a timeout while attempting to load the test libraries. For Honolulu Tech Preview 1711, all the known issues have been listed here. The return of Storage Spaces Direct is a definite takeaway. But is this build preview all that “big hyper-converged innovation” Microsoft teased us with? ReFS has been around since Windows Server 2012, and Data Deduplication is a checkbox feature for any storage device. We just have a feeling there could be bigger announcements in store.

Microsoft kicked off its highly anticipated Microsoft Connect(); 2017 annual conference on the 14th of November. This three-day annual developer conference is targeted at improving the overall developer experience for building future-oriented apps. In the words of Mitra Azizirad, Corporate VP of Microsoft’s Cloud+Enterprise : “Whether you are creating cloud native-applications, targeting the edge of devices and Internet of Things, infusing your apps with AI, or just getting started, Connect(); 2017 will equip you with the tools and skills you need to build the apps of the future” Key highlights from the Microsoft Connect(); 2017 This year the conference is all about Microsoft forming new partnerships, creating better platforms, enhancing developer productivity, and developing AI enabled tools. A large number of announcements were made pertaining to these areas. New platforms and partnerships Microsoft announced new platforms and partnerships catering to their customers as well as the open source community. Microsoft Azure + DataBricks + Apache Spark = Azure Databricks Microsoft has partnered with Databricks to bring the unique benefits of Apache Spark analytics platform with Databricks in the enterprise cloud.  Termed as Azure Databricks, this analytics platform is optimized for Azure to help data scientists, data engineers, and business decision-makers with streamlined workflows and an interactive workspace. Microsoft joins MariaDB Foundation Microsoft has collaborated with MariaDB community to work closely with the MariaDB foundation. In addition to this, they have also launched a preview of Azure Database for MariaDB service. Developers using Azure Database for MariaDB can now build intelligent apps; Azure Database for PostgreSQL and MySQL already exist. Azure Cosmos DB with Apache® Cassandra API Microsoft has also launched native support for Apache Cassandra API in Azure Cosmos DB. This comes as an integration of Azure Cosmos’s multimodal database service with Cassandra SDKs and tools, without any app code changes. This means developers can now use Cassandra-as-a-service powered by Azure Cosmos DB. GitHub Partnership on GVFS Microsoft has also partnered with Github to manage their large-scale source code repositories. This is made possible through their Git Virtual File System (GVFS) project. Microsoft has built GVFS as an open-source extension to the Git version control system, making it easy to manage over 25 million user repositories. Productivity enhancement As with every year, a key focus area has been to enhance developer productivity, at an individual as well as at a team level. For this the following announcements were made: Azure DevOps Projects Microsoft announced their Azure DevOps project. This will allow developers to build an Azure application on any Azure service using a wide variety of tech stacks. It can also configure a full DevOps pipeline fueled by Visual Studio Team Services. Visual Studio App Center Microsoft has also announced the general availability of its Visual Studio App Center. This app development lifecycle solution helps developers automate, test, manage, distribute, and monitor the lifecycle of their iOS, Android, Windows and macOS apps in the cloud. Visual Studio Live Share Microsoft also unveiled a real-time collaboration tool for developer productivity enhancement. Termed as Visual Studio Live Share, it allows developers using Visual Studio or Visual Studio Code to collaboratively edit and debug their code in real time. It also allows sharing their projects with other developers. Visual Studio Connected Environment for Azure Container Service (AKS) Developers can now use a new connected environment on Microsoft. This would be offered by Azure Container Services(AKS). It would allow developers to easily edit and debug cloud-native applications working on Kubernetes. New Artificial Intelligence tools Artificial Intelligence is revolutionizing how humans interact with technology. With this in mind, Microsoft has announced new AI tools to bring machine learning and intelligence to its developer audience. Azure IoT Edge Microsoft has made available the preview of Azure IoT Edge, a service for building AI applications for the Edge.  Support for AI Toolkit for Azure IoT Edge, Azure Machine Learning, Azure Stream Analytics and Azure Functions is also provided.  Developers can easily build AI applications using Azure Machine Learning and then deploy and manage them on the Azure IoT Edge. Visual Studio Tools for AI Visual Studio Tools for AI is an extension of their Visual Studio IDE. It will allow developers to create, debug, and edit AI applications and scale them to the cloud. It also supports popular deep learning frameworks including Cognitive Toolkit (CNTK), TensorFlow, and Caffe. Key takeaways from the Microsoft Connect(); 2017 This is what we understand from the Microsoft Connect(); 2017 announcements: Microsoft sees partnerships as the key to success, and have partnered with prominent organizations and popular open source communities to help develop better products for their consumers and improve the overall developer experience by providing them easy to use tools and services. The AI wave is a next big hit for Microsoft, as the top players in the tech world (read Google, IBM, Amazon) have already adopted AI as the weapon of choice. Microsoft is catching up real fast, with the launch of their Visual Studio platform specific to AI application. This is a good move to stand head-to-head among the leaders. Edge computing is the next cutting-edge for Microsoft, as portrayed by their Azure IoT edge service. Bringing the developer community closer, by focusing on providing developers easier ways to collaborate and share their projects. The launch of their real-time collaboration tool, Visual Studio Live Share, and a new Connected environment are the next steps towards this goal. Further announcements are expected in the upcoming days. You can visit our website for further updates on upcoming announcements and detailed analysis. For live coverage, you can tune into Connect(); 2017 for more interesting stuff from Microsoft.

Since 2010, NumPy has been supporting both Python 2 and Python 3 in parallel. But now that the Python core team is planning to discontinue Python 2 in 2020, NumPy has announced it will drop Python 2.7 support. The developers of the Python language extended support of Python 2.7 from 2015 to 2020, recognizing that many people were still using Python 2. But the transition to Python 3 was unavoidable, after all. Sensing the inevitable, NumPy decided to join the league of projects that have pledged to drop support for Python 2.7 no later than 2020. NumPy team cited “an increasing burden on our limited resources” for deciding to drop Python 2 support. “Now that we're entering the final years of community-supported Python 2, the NumPy project wants to clarify our plans, with the goal of helping our downstream ecosystem make plans and accomplish the transition with as little disruption as possible,” the project team said. NumPy’s transition plan to Python 3.x Until Dec. 31, 2018, all NumPy releases will fully support both Python 2 and Python 3. Starting Jan. 1, 2019, any new feature releases will support only Python 3. The last Python 2 supporting release will be designated as a long term support (LTS) release, meaning NumPy will continue to merge bug fixes and make bug fix releases for a longer period than usual. Specifically, it will be supported by the community until Dec. 31, 2019. “On Jan. 1, 2020, we will raise a toast to Python 2, and community support for the last Python 2 supporting release will come to an end,” NumPy announced. “However, it will continue to be available on PyPI indefinitely, and if any commercial vendors wish to extend the LTS support past this point then we are open to letting them use the LTS branch in the official NumPy repository to coordinate that.” The above is a graceful way of transition. So even though the extra 5 years were sufficient for a smooth passage, the Python team has decided to live by the core principles of free and open source software, by not obstructing third party paid support. What next? “If you are a NumPy user who requires ongoing Python 2 support in 2020 or later, then please contact your vendor. If you are a vendor who wishes to continue to support NumPy on Python 2 in 2020+, please get in touch; ideally we'd like you to get involved in maintaining the LTS before it actually hits end of life so that we can make a clean handoff,” the team said. To minimize disruption, running pip install numpy on Python 2 will continue to give the last working release in perpetuity, but after Jan. 1, 2019, it may not contain the latest features. After Jan. 1, 2020, it may not contain the latest bug fixes. Python 2.7 was a major release, and there could be users who may require third party paid support in 2020. But there is nothing like enjoying the free, first party support for any software project. Please start planning to move to Python 3.

New announcements from Microsoft Connect(); 2017, Datameer's new cloud-architected product, Visual Basic Upgrade Companion version 8.0, and more in today's trending stories in data science news. Microsoft announces new developer tools, streamlines analytics with new Azure services at Microsoft Connect(); Microsoft announces Azure Databricks powered by Apache Spark To simplify big data analytics, Microsoft announced the availability of Azure Databricks, a service designed to provide one-click setup of analytics jobs on top of Apache Spark. The service, now available in preview, intends to streamline workflows when carrying out Spark analytics, and offers a workspace for interacting with jobs. Azure Databricks has native integration with Azure SQL Data Warehouse, Azure Storage, Azure Cosmos DB, Azure Active Directory and Power BI. Databricks is a San Francisco-based startup founded by the team that developed the popular open-source Apache Spark data-processing framework at the University of California-Berkeley. Microsoft VP Frank Shaw said Azure Databricks could help in inspecting real time data patterns, “like a hotel being able to reason from structured and unstructured data, like video and sound, to discover the best type of lobby flow and check-in desk configuration for a better guest experience.” Microsoft Joins MariaDB Foundation Microsoft said it has joined MariaDB Foundation as a platinum member. The company announced the upcoming preview of Azure Database for MariaDB for a fully managed MariaDB service in the cloud. Azure Cosmos DB with Apache Cassandra API In another database upgrade on Azure, Microsoft said the Azure Cosmos DB will make data available via an API for the Cassandra NoSQL database. Access to the API is available in preview, and expands the range of data models supported by Azure Cosmos DB, which includes document, graph, key-value, table, and columnar. GitHub to adopt Microsoft's Git Virtual File System (GVFS) Microsoft and GitHub have taken their open source partnership further to extend GVFS support to GitHub. Microsoft uses the GVFS to allow it to use the Git version control system with its code for Windows, which runs to 3.5 million files weighing in at about 300GB. Git was never designed to be used with that large an amount of files, and using GVFS allows Git commands to be run on the codebase without taking an unreasonable amount of time. General availability of Visual Studio App Center Visual Studio App Center is a new cloud service for developers to ship higher-quality applications more frequently. Objective-C, Swift, Android Java, Xamarin and React Native developers can use App Center to increase productivity and accelerate application lifecycle, freeing them to spend more time on new features and better user experiences. Visual Studio Live Share Visual Studio Live Share provides unique new capability for developers to collaborate in a seamless and secure way with full project context. With this preview, developers can share projects with teammates, or other developers, to edit and debug the same code in their personalized editor or IDE. Azure DevOps Projects The preview lets developers configure a full DevOps pipeline and connect to Azure Services within five minutes for faster app development and deployment. With just a few clicks in the Azure portal, developers can set up Git repositories, wire up completely automated builds and release pipelines without any prior knowledge of how to do so. Azure IoT Edge and machine learning updates Azure IoT Edge is now available for preview, enabling AI, advanced analytics and machine learning at the Internet of Things (IoT) edge. Such an integration with Azure IoT Edge and AI deployment on iOS devices with Core ML can bring AI everywhere from the cloud to the IoT edge of devices. Azure SQL Database machine learning services preview Microsoft announced new support for R models inside SQL Database making it seamless for data scientists to develop and train models in Azure Machine Learning and deploy those models directly to Azure SQL Database to create predictions at blazing-fast speeds. Announcing VBUC 8.0 Visual Basic Upgrade Companion 8.0 comes with additional machine learning Mobilize.Net has announced Visual Basic Upgrade Companion version 8.0. The new release enables faster migrations, better code generation, and additional machine learning. Customers can try VBUC 8.0 at https://www.mobilize.net/vbuc-free-trial. “Old versions of Visual Basic are unable to support agile, fast development, but these apps are still performing critical roles in organizations,” said Tom Button, CEO of Mobilize.Net. “Mobilize.Net products provide an onramp to DevOps and Agile development helping customers to move to the latest technologies and methodologies.” Big Data platforms in data science news Datameer introduces cloud-architected big data platform through AWS Datameer has announced a new edition of their data preparation and exploration software that is architected specifically for the cloud, and delivered through Amazon Web Services (AWS).The new product comes in response to demands from businesses that are moving to a hybrid cloud-computing model and seeking data preparation tools that are more agile and scalable. While Datameer's software has been available as a hosted service on cloud platforms, the new release is specifically configured for the cloud. The main difference is that the new software's compute and storage functionality is separated, making the software more flexible by better utilizing compute and storage resources. John Morrell, Datameer's senior director of product marketing, said the new product will help businesses resolve the dilemma of working with "shadow data lakes" – massive stores of data in cloud systems such as AWS' S3 (Simple Storage Service) that are difficult to process and analyze. Hitachi announces new 100x faster technology for open source software based big data analytics Hitachi has developed a technology increasing the speed of big data analytics on an open source software Hadoop-based distributed data processing platform. The new technology speeds up the analytics by up to 100 times that of a conventional system. This technology converts data processing procedure generated for software processing in conventional Hadoop data processing to that optimized for parallel processing on hardware, thus enabling high-speed processing of various types of data in FPGA. As a result, less number of servers will be needed when conducting high-speed big data analytics, thus minimizing IT investment while enabling interactive analytics by data scientists, quick on-site business decision making, and other timely information services. This technology will be applied to areas such as finance and communication, Hitachi said. Cryptocurrency in News Bitcoin Gold Launched The latest fork in the bitcoin blockchain, Bitcoin Gold, has gone live. Developers have released the necessary code on GitHub. They aim to create the largest decentralized community in the crypto world.

Amazon is gearing up its cloud game against Microsoft and Google with the launch of Project Ironman. According to a report by Information, Amazon is working on its artificial intelligence and machine learning capabilities with the introduction of a new cloud-based service. Code-named as Ironman, it is officially expected to be announced at the AWS re:Invent 2017 scheduled to begin in the last week of November. Ironman is the alliance of three prominent attractions this year: data warehouses, artificial intelligence, and cloud. Data warehouses are databases with faster data retrieval and analytic capabilities. And it seems pretty clear that the cloud market is a highly lucrative space for prominent organizations. More and more cloud vendors seek to rise above their competitors with A.I. as their weapon of choice. With Google being the leader in this race, Amazon seems to be catching up real fast. Although Amazon already offers cloud solutions, Ironman comes as an upgrade which caters specifically to machine learning and general artificial intelligence companies who are looking for high-performance cloud-based services. How will it do that? Ironman will include a new AWS cloud which will aid in the collection of massive volumes of data from multiple sources and then store them in a centralized location, accessible through end-user queries. As of now, Ironman will focus on companies in insurance, energy, fraud detection and drug discovery. It will also be beneficial to data scientists, ML developers, and engineers in developing A.I. applications. The Ironman services will be compatible with NVIDIA GPU chips as well as with the field programmable gate array chips (FPGA). These chips can be reprogrammed depending on the kind of software needed. DataRobot, an automation startup and Domino Data Lab, an organization developing machine learning solutions for data scientists are two organizations reportedly working with Amazon on the Ironman project. Ironman is expected to be Amazon’s most capable solution for processing machine learning related data. It is expected to be a direct rival to the Google Cloud and Microsoft Azure, the two popular alternatives. The Information also reported that AWS plans to unveil a cloud-based deep learning service at their event. It is meant to directly compete with Google’s TensorFlow. These two upcoming platforms should build on Amazon’s existing reputation in the A.I. market and gain a superior standing among its competitors in the cloud and A.I. space. More details regarding Ironman are expected to be announced at re:Invent.

Microsoft recently announced the stirring combination of Apache Spark Analytics platform and the Azure cloud at the Microsoft Connect();. Presenting, Azure Databricks! Azure Databricks is a close collaboration between Microsoft and Databricks to bring about benefits not present in any other cloud platforms. Azure Databricks: The trinity effect This is the very first time that an Apache Spark platform provider ’Databricks’ has partnered with a cloud provider ‘Microsoft Azure’ to bring about a highly optimized platform for data analytics workloads. Data management on the cloud has opened up pathways in the field of Artificial Intelligence, predictive analytics, and for real-time applications. Apache Spark has been everyone’s favorite platform to implement these cutting-edge analytics applications, due to its vast community and a worldwide enterprise network. Its ability to run powerful analytics algorithms at scale, allows businesses to derive real-time insights with ease. However, the management and deployment of Spark within the enterprise use cases, which includes a large number of users and has strong security requirements, was a bit challenging. Azure Databricks comes as a solution to this, by providing business users with a platform to work effectively with the data professionals --data scientists and data engineers. Benefits of Azure Databricks: A sneak-peek   Highly optimized for a cost-efficient and improved performance in the cloud, with an added end-to-end, managed Apache Spark platform. Includes features such as one-click deployment, autoscaling, and an optimized Databricks Runtime that can improve the performance of Spark jobs in the cloud by 10-100x. A simple and cost-efficient implementation of large-scale Spark workloads. Includes an interactive notebook environment, along with a few monitoring tools, and security controls that make it easy to leverage Spark in enterprises with a huge number of users. Optimized connectors to Azure storage platforms (e.g. Data Lake and Blob Storage) for fast data access. A one-click management directly from the Azure console. It even includes common analytics libraries, such as the Python and R data science stacks, pre-installed to use them with Spark in order to derive insights. The partnership Architecture: Source: https://azure.microsoft.com/en-us/blog/a-technical-overview-of-azure-databricks/ Azure Databricks has an architecture which allows customers to effectively and easily connect Azure Databricks to any of the storage resource present in their account. For instance, an existing subscription of  Blob store or Data Lake. Also, the Databricks is centrally managed through the Azure control center. Hence, it requires no additional setup. Fully integrated Azure features Azure Databricks has been appended to the best of Microsoft Azure features. Some of them are listed below: A secure and private data control where the ownership rights are with the customer alone Diversity in the network infrastructure needs Integration of the Azure Storage and Azure Data Lake An Azure Active Directory, which provides control of access to resources used An integration of the Azure SQL Data Warehouse, Azure SQL DB, and Azure CosmosDB The latest generation of Azure hardware (Dv3 VMs), with NvMe SSDs with 100us latency on IO, making Databricks I/O performance much better Azure has many other features that have been integrated into the Azure Databricks. For a more detailed overview of Azure Databricks you can visit the official link here.

Google has made it quite accessible for people to collaborate their documents, spreadsheets, and so on, with the Google Drive feature. What next? If you are one of those data science nerds who love coding, this roll-out from Google would be an amazing experimental ground for you. Google released its coLaboratory project, a new tool, and a boon for data science and analysis. It is designed in a way to make collaborating on data easier; similar to a Google document. This means it is capable of running code and providing simultaneous output within the document itself. Collaboration is what sets coLaboratory apart. It allows an improved collaboration among people having distinct skill sets--one may be great at coding, while the other might be well aware of the front-end or GUI aspects of the project. Just as you store and share a Google document or spreadsheets, you can store and share code with coLaboratory notebooks, in Google Drive. All you have to do is, click on the 'Share' option at the top right of any coLaboratory notebook. You can also look up to the Google Drive file sharing instructions. Thus, it sets new improvements for the ad-hoc workflows without the need of mailing documents back and forth. CoLaboratory includes a Jupyter notebook environment that does not require any setup for using it. With this, one does not need to download, install, or run anything on their computer. All they would need is, just a browser and they can use and share Jupyter notebooks. At present, coLaboratory functions with Python 2.7 on the desktop version of Chrome only. The reason for this is, coLab with Python 2.7 has been an internal tool for Google, for many years. Although, making it available on other browsers and with an added support for other Jupyter Kernels such as R or Scala is on the cards, soon. CoLaboratory’s GitHub repository contains two dependent tools, which one can make use of to leverage the tool onto the browser. First is the coLaboratory Chrome App and the other is coLaboratory with Classic Jupyter Kernels.  Both tools can be used for creating and storing notebooks within Google Drive. This allows a collaborative editing within the notebooks. The only difference is that Chrome App executes all the code within its browser using the PNaCl Sandbox. Whereas, the CoLaboratory classic code execution is done using the local Jupyter kernels (IPython kernel) that have a complete access to the host systems and files. The coLaboratory Chrome App aids in setting up a collaborative environment for data analysis. This can be a hurdle at times, as requirements vary among different machines and operating systems. Also, the installation errors can be cryptic too. However, just with a single click, coLaboratory, IPython and a large set of popular scientific python libraries can be installed. Also, because of the Portable Native Client (PNaCl), coLaboratory is secure and runs at local speeds. This allows new users to set out on exploring IPython at a faster speed. Here’s what coLaboratory brings about for the code-lovers: No additional installation required the browser does it all The capabilities of coding now within a document Storing and sharing the notebooks on Google Drive Real-time collaboration possible; no fuss of mailing documents to and fro You can find a detailed explanation of the tool on GitHub.