How-To Tutorials - Programming

A design pattern is a reusable solution to a recurring problem. The term is really broad in its definition and can span multiple domains of an application. However, the term is often associated with a well-known set of object-oriented patterns that were popularized in the 90s by the book, Design Patterns: Elements of Reusable Object- Oriented Software, Pearson Education, by the almost legendary Gang of Four (GoF): Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. This article is an excerpt from the book Node.js Design Patterns, Third Edition by Mario Casciaro and Luciano Mammino – a comprehensive guide for learning proven patterns, techniques, and tricks to take full advantage of the Node.js platform. In this article, we’ll look at the behavior of components in software design. We’ll learn how to combine objects and how to define the way they communicate so that the behavior of the resulting structure becomes extensible, modular, reusable, and adaptable. After introducing all the behavioral design patterns, we will dive deep into the details of the strategy pattern. Now, it's time to roll up your sleeves and get your hands dirty with some behavioral design patterns. Types of Behavioral Design Patterns The Strategy pattern allows us to extract the common parts of a family of closely related components into a component called the context and allows us to define strategy objects that the context can use to implement specific behaviors. The State pattern is a variation of the Strategy pattern where the strategies are used to model the behavior of a component when under different states. The Template pattern, instead, can be considered the "static" version of the Strategy pattern, where the different specific behaviors are implemented as subclasses of the template class, which models the common parts of the algorithm. The Iterator pattern provides us with a common interface to iterate over a collection. It has now become a core pattern in Node.js. JavaScript offers native support for the pattern (with the iterator and iterable protocols). Iterators can be used as an alternative to complex async iteration patterns and even to Node.js streams. The Middleware pattern allows us to define a modular chain of processing steps. This is a very distinctive pattern born from within the Node.js ecosystem. It can be used to preprocess and postprocess data and requests. The Command pattern materializes the information required to execute a routine, allowing such information to be easily transferred, stored, and processed. The Strategy Pattern The Strategy pattern enables an object, called the context, to support variations in its logic by extracting the variable parts into separate, interchangeable objects called strategies. The context implements the common logic of a family of algorithms, while a strategy implements the mutable parts, allowing the context to adapt its behavior depending on different factors, such as an input value, a system configuration, or user preferences. Strategies are usually part of a family of solutions and all of them implement the same interface expected by the context. The following figure shows the situation we just described: Figure 1: General structure of the Strategy pattern Figure 1 shows you how the context object can plug different strategies into its structure as if they were replaceable parts of a piece of machinery. Imagine a car; its tires can be considered its strategy for adapting to different road conditions. We can fit winter tires to go on snowy roads thanks to their studs, while we can decide to fit high-performance tires for traveling mainly on motorways for a long trip. On the one hand, we don't want to change the entire car for this to be possible, and on the other, we don't want a car with eight wheels so that it can go on every possible road. The Strategy pattern is particularly useful in all those situations where supporting variations in the behavior of a component requires complex conditional logic (lots of if...else or switch statements) or mixing different components of the same family. Imagine an object called Order that represents an online order on an e-commerce website. The object has a method called pay() that, as it says, finalizes the order and transfers the funds from the user to the online store. To support different payment systems, we have a couple of options: Use an ..elsestatement in the pay() method to complete the operation based on the chosen payment option Delegate the logic of the payment to a strategy object that implements the logic for the specific payment gateway selected by the user In the first solution, our Order object cannot support other payment methods unless its code is modified. Also, this can become quite complex when the number of payment options grows. Instead, using the Strategy pattern enables the Order object to support a virtually unlimited number of payment methods and keeps its scope limited to only managing the details of the user, the purchased items, and the relative price while delegating the job of completing the payment to another object. Let's now demonstrate this pattern with a simple, realistic example. Multi-format configuration objects Let's consider an object called Config that holds a set of configuration parameters used by an application, such as the database URL, the listening port of the server, and so on. The Config object should be able to provide a simple interface to access these parameters, but also a way to import and export the configuration using persistent storage, such as a file. We want to be able to support different formats to store the configuration, for example, JSON, INI, or YAML. By applying what we learned about the Strategy pattern, we can immediately identify the variable part of the Config object, which is the functionality that allows us to serialize and deserialize the configuration. This is going to be our strategy. Creating a new module Let's create a new module called config.js, and let's define the generic part of our configuration manager: import { promises as fs } from 'fs' import objectPath from 'object-path' export class Config { constructor (formatStrategy) {                           // (1) this.data = {} this.formatStrategy = formatStrategy } get (configPath) {                                       // (2) return objectPath.get(this.data, configPath) } set (configPath, value) {                                // (2) return objectPath.set(this.data, configPath, value) } async load (filePath) {                                  // (3) console.log(`Deserializing from ${filePath}`) this.data = this.formatStrategy.deserialize( await fs.readFile(filePath, 'utf-8') ) } async save (filePath) {                                  // (3) console.log(`Serializing to ${filePath}`) await fs.writeFile(filePath, this.formatStrategy.serialize(this.data)) } } This is what's happening in the preceding code: In the constructor, we create an instance variable called data to hold the configuration data. Then we also store formatStrategy, which represents the component that we will use to parse and serialize the data. We provide two methods, set()and get(), to access the configuration properties using a dotted path notation (for example, property.subProperty) by leveraging a library called object-path (nodejsdp.link/object-path). The load() and save() methods are where we delegate, respectively, the deserialization and serialization of the data to our strategy. This is where the logic of the Config class is altered based on the formatStrategy passed as an input in the constructor. As we can see, this very simple and neat design allows the Config object to seamlessly support different file formats when loading and saving its data. The best part is that the logic to support those various formats is not hardcoded anywhere, so the Config class can adapt without any modification to virtually any file format, given the right strategy. Creating format Strategies To demonstrate this characteristic, let's now create a couple of format strategies in a file called strategies.js. Let's start with a strategy for parsing and serializing data using the INI file format, which is a widely used configuration format (more info about it here: nodejsdp.link/ini-format). For the task, we will use an npm package called ini (nodejsdp.link/ini): import ini from 'ini' export const iniStrategy = { deserialize: data => ini.parse(data), serialize: data => ini.stringify(data) } Nothing really complicated! Our strategy simply implements the agreed interface, so that it can be used by the Config object. Similarly, the next strategy that we are going to create allows us to support the JSON file format, widely used in JavaScript and in the web development ecosystem in general: export const jsonStrategy = { deserialize: data => JSON.parse(data), serialize: data => JSON.stringify(data, null, '  ') } Now, to show you how everything comes together, let's create a file named index.js, and let's try to load and save a sample configuration using different formats: import { Config } from './config.js' import { jsonStrategy, iniStrategy } from './strategies.js' async function main () { const iniConfig = new Config(iniStrategy) await iniConfig.load('samples/conf.ini') iniConfig.set('book.nodejs', 'design patterns') await iniConfig.save('samples/conf_mod.ini') const jsonConfig = new Config(jsonStrategy) await jsonConfig.load('samples/conf.json') jsonConfig.set('book.nodejs', 'design patterns') await jsonConfig.save('samples/conf_mod.json') } main() Our test module reveals the core properties of the Strategy pattern. We defined only one Config class, which implements the common parts of our configuration manager, then, by using different strategies for serializing and deserializing data, we created different Config class instances supporting different file formats. The example we've just seen shows us only one of the possible alternatives that we had for selecting a strategy. Other valid approaches might have been the following: Creating two different strategy families: One for the deserialization and the other for the serialization. This would have allowed reading from a format and saving to another. Dynamically selecting the strategy: Depending on the extension of the file provided; the Config object could have maintained a map extension → strategy and used it to select the right algorithm for the given extension. As we can see, we have several options for selecting the strategy to use, and the right one only depends on your requirements and the tradeoff in terms of features and the simplicity you want to obtain. Furthermore, the implementation of the pattern itself can vary a lot as well. For example, in its simplest form, the context and the strategy can both be simple functions: function context(strategy) {...} Even though this may seem insignificant, it should not be underestimated in a programming language such as JavaScript, where functions are first-class citizens and used as much as fully-fledged objects. Between all these variations, though, what does not change is the idea behind the pattern; as always, the implementation can slightly change but the core concepts that drive the pattern are always the same. Summary In this article, we dive deep into the details of the strategy pattern, one of the Behavioral Design Patterns in Node.js. Learn more in the book, Node.js Design Patterns, Third Edition by Mario Casciaro and Luciano Mammino. About the Authors Mario Casciaro is a software engineer and entrepreneur. Mario worked at IBM for a number of years, first in Rome, then in Dublin Software Lab. He currently splits his time between Var7 Technologies-his own software company-and his role as lead engineer at D4H Technologies where he creates software for emergency response teams. Luciano Mammino wrote his first line of code at the age of 12 on his father's old i386. Since then he has never stopped coding. He is currently working at FabFitFun as principal software engineer where he builds microservices to serve millions of users every day.

JMeter is a 100% pure Java desktop application. JMeter is found to be very useful and convenient in support of functional testing. Although JMeter is known more as a performance testing tool, functional testing elements can be integrated within the Test Plan, which was originally designed to support load testing. Many other load-testing tools provide little or none of this feature, restricting themselves to performance-testing purposes. Besides integrating functional-testing elements along with load-testing elements in the Test Plan, you can also create a Test Plan that runs these exclusively. In other words, aside from creating a Load Test Plan, JMeter also allows you to create a Functional Test Plan. This flexibility is certainly resource-efficient for the testing project. In this article by Emily H. Halili, we will give you a walkthrough on how to create a Test Plan as we incorporate and/or configure JMeter elements to support functional testing. Preparing for Functional Testing JMeter does not have a built-in browser, unlike many functional-test tools. It tests on the protocol layer, not the client layer (i.e. JavaScripts, applets, and many more.) and it does not render the page for viewing. Although, by default that embedded resources can be downloaded, rendering these in the Listener | View Results Tree may not yield a 100% browser-like rendering. In fact, it may not be able to render large HTML files at all. This makes it difficult to test the GUI of an application under testing. However, to compensate for these shortcomings, JMeter allows the tester to create assertions based on the tags and text of the page as the HTML file is received by the client. With some knowledge of HTML tags, you can test and verify any elements as you would expect them in the browser. It is unnecessary to select a specific workload time to perform a functional test. In fact, the application you want to test may even reside locally, with your own machine acting as the "localhost" server for your web application. For this article, we will limit ourselves to selected functional aspects of the page that we seek to verify or assert. Using JMeter Components We will create a Test Plan in order to demonstrate how we can configure the Test Plan to include functional testing capabilities. The modified Test Plan will include these scenarios: 1. Create Account —New Visitor creating an Account 2. Login User —User logging in to an Account Following these scenarios, we will simulate various entries and form submission as a request to a page is made, while checking the correct page response to these user entries. We will add assertions to the samples following these scenarios to verify the 'correctness' of a requested page. In this manner, we can see if the pages responded correctly to invalid data. For example, we would like to check that the page responded with the correct warning message when a user enters an invalid password, or whether a request returns the correct page. First of all, we will create a series of test cases following the various user actions in each scenario. The test cases may be designed as follows: CREATE ACCOUNT Test Steps Data Expected 1 Go to Home page. www.packtpub.com Home page loads and renders with no page error 2 Click Your Account link (top right). User action 1. Your Account page loads and renders with no page error.2. Logout link is not found. 3 No Password: - Enter email address in Email text field.- Click the Create Account and Continue button. email=EMAIL 1. Your Account page resets with Warning message-Please enter password.2. Logout link not found. 4 Short Password: - Enter email address in Email text field.- Enter password in Password text field.- Enter password in Confirm Password text field. - Click Create Account and Continue button. email=EMAILpassword=SHORT_PWD confirm password=SHORT_PWD 1. Your Account page resets with Warning message-Your password must be 8 characters or longer.2. Logout link is not found. 5 Unconfirmed Password: - Enter email address in Email text field.- Enter password in Password text field.- Enter password in Confirm Password text field. - Click Create Account and Continue button. email=EMAILpassword=VALID_PWDconfirm password=INVALID_PWD 1. Your Account page resets with Warning messagePassword does not match.2. Logout link is not found. 6 Register Valid User: - Enter email address in Email text field.- Enter password in Password text field.- Enter password in Confirm Password text field. - Click Create Account and Continue button. email=EMAILpassword=VALID_PWDconfirm password=VALID_PWD 1. Logout link is found.2. Page redirects to User Account page.3. Message found: You are registered as: e:<EMAIL>. 7 Click Logout link. User action 1. Logout link is NOT found.     LOGIN USER Test Steps Data Expected 1 Click Home page. User action 1. WELCOME tab is active. 2 Log in Wrong Password: - Enter email in Email text field- Enter password at Password text field.- Click Login button. email=EMAILpassword=INVALID_PWD 1. Logout link is NOT found.2. Page refreshes.3. Warning message-Sorry your password was incorrect appears. 3 Log in Non-Exist Account:- Enter email in Email text field.- Enter password in Password text field.- Click Login button. email=INVALID_EMAILpassword=INVALID_PWD 1. Logout link is NOT found.2. Page refreshes.3. Warning message-Sorry, this does not match any existing accounts. Please check your details and try again or open a new account below appears. 4 Log in Valid Account:- Enter email in Email text field.- Enter password in Password text field.- Click Login-button. email=EMAILpassword=VALID_PWD 1. Logout link is found.2. Page reloads.3. Login successful message-You are logged in as: appears. 5 Click Logout link. User action 1. Logout link is NOT found.    

(For more resources related to this topic, see here.) Testing Backbone applications is no different than testing any other application; you are still going to drive your code from your specs, except that Backbone is already leveraging a lot of functionality for you, for free. So expect to write less code, and consequently less specs. Testing Backbone Views We already have seen some of the advantages of using the View pattern in Testing Frontend Code, and are already creating our interface components in such a manner. So how can a Backbone View be different from what we have done so far? It retains a lot of the patterns that we have discussed as best practices for creating maintainable browser code, but with some syntax sugar and automation to make our life easier. They are the glue code between the HTML and the model, and the Backbone View's main responsibility is to add behavior to the interface, while keeping it in sync with a model or collection. As we will see, Backbone's biggest triumph is how it makes an easy-to-handle DOM event delegation, a task usually done with jQuery. Declaring a new View Declaring a new View is going to be a matter of extending the base Backbone.View object. To demonstrate how it works we need an example. We are going to create a new View and its responsibility is going to be to render a single investment on the screen. We are going to create it in such a way that allows its use by the InvestmentListView component. This is a new component and spec, written in src/InvestmentView.js and spec/InvestmentViewSpec.js respectively. In the spec file, we can write something similar to this: describe("InvestmentView", function() { var view; beforeEach(function() { view = new InvestmentView(); }); it("should be a Backbone View", function() { expect(view).toEqual(jasmine.any(Backbone.View)); });}); Which translates into an implementation that extends the base Backbone View component: (function (Backbone) { var InvestmentView = Backbone.View.extend() this.InvestmentView = InvestmentView;})(Backbone); And now we are ready to explore some of the new functionalities provided by Backbone. The el property Like the View pattern a Backbone View also has an attribute containing the reference to its DOM element. The difference here is that Backbone comes with it by default, providing: view.el: The DOM element view.$el: The jQuery object for that element view.$: A scoped jQuery lookup function (the same way we have implemented) And if you don't provide an element on the constructor, it creates an element for you automatically. Of course the element it creates is not attached to the document, and is up to the View's user code to attach it. Here is a common pattern you see while using Views: Instantiate it: var view = new InvestmentView(); Call the render function to draw the View's components view.render() Append its element to the page document: $('body').append(view.el); Given our clean implementation of the InvestmentView, if you would go ahead and execute the preceding code on a clean page, you would get the following result: <body> <div></div></body> An empty div element; that is the default element created by Backbone. But we can change that with a few configuration parameters on the InvestmentView declaration. Let's say we want the DOM element of InvestmentView to be a list item (li) with an investment CSS class. We could write this spec using the familiar Jasmine jQuery matchers: describe("InvestmentView", function() { var view; beforeEach(function() { view = new InvestmentView(); }); it("should be a list item with 'investment' class", function() { expect(view.$el).toBe('li.investment'); });}); You can see that we didn't use the setFixtures function, since we can run this test against the element instance available on the View. Now to the implementation; all we have to do, is define two simple attributes in the View definition, and Backbone will use them to create the View's element: var InvestmentView = Backbone.View.extend({ className: 'investment', tagName: 'li'}); By looking at the implementation you might be wondering if we shouldn't test it. Here I would recommend against it, since you wouldn't get any benefit from that approach, as this spec is much more solid. That is great, but how do we add content to that DOM element? That is up to the render function we are going to see next. var view = new InvestmentView({ el: $('body') }); But by letting the View handle its rendering, we get better componentization and we can also gain on performance. Rendering Now that we understand that it is a good idea to have an empty element available on the View, we must get into the details of how to draw on this empty canvas. Backbone Views already come with an available render function, but it is a dummy implementation, so it is up to you to define how it works. Going back to the InvestmentView example, let's add a new acceptance criterion to describe how it should be rendered. We are going to start by expecting that it renders the return of investment as a percentage value. Here is the spec implementation: describe("InvestmentView", function() { var view, investment; beforeEach(function() { investment = new Investment(); view = new InvestmentView({ model: investment }); }); describe("when rendering", function() { beforeEach(function() { investment.set('roi', 0.1); view.render(); }); it("should render the return of investment", function() { expect(view.$el).toContainHtml('10%'); }); });}); That is a very standard spec with concepts that we have seen before and the implementation is just a matter of defining the render function on the InvestmentView declaration: var InvestmentView = Backbone.View.extend({ className: 'investment', tagName: 'li', render: function () { this.$el.html('<p>'+ formatedRoi.call(this) +'<p>'); return this; }});function formatedRoi () { return (this.model.get('roi') * 100) + '%';} It is using the this.$el property to add some HTML content to the View's element. There are some details that are important for you to notice regarding the render function implementation: We are using the jQuery.html function, so that we can invoke the render function multiple times without duplicating the View's content. The render function returns the View instance once it has completed rendering. This is a common pattern to allow chained calls, such as: $('body').append(new InvestmentView().render().el); Now back to the test. You can see that we weren't testing for the specific HTML snippet, but rather, that just 10 percent text was rendered. You could have done a more thoroughly written spec by checking the exact same HTML at the expectation, but that ends up adding test complexity with little benefit. Summary In this article, you have seen how to use Backbone to do some heavy lifting, allowing you to focus more on your application code. I showed you the power of events, and how they make integration between different components much easier, allowing you to keep your models and Views in sync. Resources for Article: Further resources on this subject: The architecture of JavaScriptMVC [Article] Working with JavaScript in Drupal 6: Part 1 [Article] Syntax Validation in JavaScript Testing [Article]

SQL Server Compact Edition 3.5 can be used to create applications that are useful for a number of business uses such as: Portable applications; Occasionally connected clients and embedded applications and devices. SQL Server Compact differs from other SQL Servers in that there is just one file which can be password protected and features 128-bit file level encryption. It is referential integrity compliant; supports multiple connections; has transactions support with rich data types. In this tutorial by Jayaram Krishnaswamy, various scenarios where you may need to connect to SQL Server Compact using Visual Studio IDE (both 2008 and 2010) are described in detail. Connecting to SQL Server Compact 4.5 using Visual Studio 2010 Express (free version of Visual Studio) is also described. The connection is the starting point for any database related program and therefore mastering the connection task is crucial to work with the SQL Server Compact. (For more resources on Microsoft, see here.) If you are familiar with SQL Server you already know much of SQL Server Compact. It can be administered from SSMS and, using SQL Syntax and ADO.NET technology you can be immediately productive with SQL Server Compact.It is free to download (also free to deploy and redistribute) and comes in the form of just one code-free file. Its small foot print makes it easily deployable to a variety of device sizes and requires no administration. It also supports a subset of T-SQL and a rich set of data types. It can be used in creating desktop/web applications using Visual Studio 2008 and Visual Studio 2010. It also comes with a sample Northwind database. Download details Microsoft SQL Server Compact 3.5 may be downloaded from this site here. Make sure you download detailed features of this program from the same site. Also several bugs have been fixed in the program as detailed in the two SP's. Link to the latest service pack SP2 is here. By applying SP2 the installed version on the machine is upgraded to the latest version. Connecting to SQL Server Compact from Windows and Web projects You can use the Server Explorer in Visual Studio to drag and drop objects from SQL Server Compact provided you add a connection to the SQL Server Compact. In fact, in Visual Studio 2008 IDE you can configure a data connection without even starting a project from the View menu as shown here. When you click Add Connection... the following window will be displayed. This brings up the Add Connection dialog shown here. Click Change... to choose the correct data source for SQL Server Compact. The default is SQL Server client. The Change Data Source window is displayed as shown. Highlight Microsoft SQL Server Compact 3.5 and click OK. You are returned to Add Connection where you can browse or create a database or, choose also from a ActiveSync connected device such as a Smart phone which has a SQL Server Compact for devices installed. Presently connect to one on the computer (My Computer default option)-the sample database Northwind. Click Browse.... The Select SQL Server Compact 3.5 Database File dialog opens where your sample database Northwind is displayed as shown. Click Open. The database file is entered in the Add Connection dialogue. You may test the connection. You should get a Test connection succeeded message from Microsoft Visual Studio. Click OK. The Northwind.sdf file is displayed as a tree with Tables and View as shown in the next figure. Right click Northwind.sdf in the Server Explorer above and click Properties drop-down menu item. You will see the connection string for this conneciton as shown here.

 In this article by Umesh R Sharma, author of the book Practical Microservices, we will cover API Gateway and its need with simple and short examples. (For more resources related to this topic, see here.) Dynamic websites show a lot on a single page, and there is a lot of information that needs to be shown on the page. The common success order summary page shows the cart detail and customer address. For this, frontend has to fire a different query to the customer detail service and order detail service. This is a very simple example of having multiple services on a single page. As a single microservice has to deal with only one concern, in result of that to show much information on page, there are many API calls on the same page. So, a website or mobile page can be very chatty in terms of displaying data on the same page. Another problem is that, sometimes, microservice talks on another protocol, then HTTP only, such as thrift call and so on. Outer consumers can't directly deal with microservice in that protocol. As a mobile screen is smaller than a web page, the result of the data required by the mobile or desktop API call is different. A developer would want to give less data to the mobile API or have different versions of the API calls for mobile and desktop. So, you could face a problem such as this: each client is calling different web services and keeping track of their web service and developers have to give backward compatibility because API URLs are embedded in clients like in mobile app. Why do we need the API Gateway? All these preceding problems can be addressed with the API Gateway in place. The API Gateway acts as a proxy between the API consumer and the API servers. To address the first problem in that scenario, there will only be one call, such as /successOrderSummary, to the API Gateway. The API Gateway, on behalf of the consumer, calls the order and user detail, then combines the result and serves to the client. So basically, it acts as a facade or API call, which may internally call many APIs. The API Gateway solves many purposes, some of which are as follows. Authentication API Gateways can take the overhead of authenticating an API call from outside. After that, all the internal calls remove security check. If the request comes from inside the VPC, it can remove the check of security, decrease the network latency a bit, and make the developer focus more on business logic than concerning about security. Different protocol Sometimes, microservice can internally use different protocols to talk to each other; it can be thrift call, TCP, UDP, RMI, SOAP, and so on. For clients, there can be only one rest-based HTTP call. Clients hit the API Gateway with the HTTP protocol and the API Gateway can make the internal call in required protocol and combine the results in the end from all web service. It can respond to the client in required protocol; in most of the cases, that protocol will be HTTP. Load-balancing The API Gateway can work as a load balancer to handle requests in the most efficient manner. It can keep a track of the request load it has sent to different nodes of a particular service. Gateway should be intelligent enough to load balances between different nodes of a particular service. With NGINX Plus coming into the picture, NGINX can be a good candidate for the API Gateway. It has many of the features to address the problem that is usually handled by the API Gateway. Request dispatching (including service discovery) One main feature of the gateway is to make less communication between client and microservcies. So, it initiates the parallel microservices if that is required by the client. From the client side, there will only be one hit. Gateway hits all the required services and waits for the results from all services. After obtaining the response from all the services, it combines the result and sends it back to the client. Reactive microservice designs can help you achieve this. Working with service discovery can give many extra features. It can mention which is the master node of service and which is the slave. Same goes for DB in case any write request can go to the master or read request can go to the slave. This is the basic rule, but users can apply so many rules on the basis of meta information provided by the API Gateway. Gateway can record the basic response time from each node of service instance. For higher priority API calls, it can be routed to the fastest responding node. Again, rules can be defined on the basis of the API Gateway you are using and how it will be implemented. Response transformation Being a first and single point of entry for all API calls, the API Gateway knows which type of client is calling a mobile, web client, or other external consumer; it can make the internal call to the client and give the data to different clients as per needs and configuration. Circuit breaker To handle the partial failure, the API Gateway uses a technique called circuit breaker pattern. A service failure in one service can cause the cascading failure in the flow to all the service calls in stack. The API Gateway can keep an eye on some threshold for any microservice. If any service passes that threshold, it marks that API as open circuit and decides not to make the call for a configured time. Hystrix (by Netflix) served this purpose efficiently. Default value in this is failing of 20 requests in 5 seconds. Developers can also mention the fall back for this open circuit. This fall back can be of dummy service. Once API starts giving results as expected, then gateway marks it as a closed service again. Pros and cons of API Gateway Using the API Gateway itself has its own pros and cons. In the previous section, we have described the advantages of using the API Gateway already. I will still try to make them in points as the pros of the API Gateway. Pros Microservice can focus on business logic Clients can get all the data in a single hit Authentication, logging, and monitoring can be handled by the API Gateway Gives flexibility to use completely independent protocols in which clients and microservice can talk It can give tailor-made results, as per the clients needs It can handle partial failure Addition to the preceding mentioned pros, some of the trade-offs are also to use this pattern. Cons It can cause performance degrade due to lots of happenings on the API Gateway With this, discovery service should be implemented Sometimes, it becomes the single point of failure Managing routing is an overhead of the pattern Adding additional network hope in the call Overall. it increases the complexity of the system Too much logic implementation in this gateway will lead to another dependency problem So, before using the API Gateway, both of the aspects should be considered. Decision of including the API Gateway in the system increases the cost as well. Before putting effort, cost, and management in this pattern, it is recommended to analysis how much you can gain from it. Example of API Gateway In this example, we will try to show only sample product pages that will fetch the data from service product detail to give information about the product. This example can be increased in many aspects. Our focus of this example is to only show how the API Gateway pattern works; so we will try to keep this example simple and small. This example will be using Zuul from Netflix as an API Gateway. Spring also had an implementation of Zuul in it, so we are creating this example with Spring Boot. For a sample API Gateway implementation, we will be using http://start.spring.io/ to generate an initial template of our code. Spring initializer is the project from Spring to help beginners generate basic Spring Boot code. A user has to set a minimum configuration and can hit the Generate Project button. If any user wants to set more specific details regarding the project, then they can see all the configuration settings by clicking on the Switch to the full version button, as shown in the following screenshot: Let's create a controller in the same package of main application class and put the following code in the file: @SpringBootApplication @RestController public class ProductDetailConrtoller { @Resource ProductDetailService pdService; @RequestMapping(value = "/product/{id}") public ProductDetail getAllProduct( @PathParam("id") String id) { return pdService.getProductDetailById(id); } }   In the preceding code, there is an assumption of the pdService bean that will interact with Spring data repository for product detail and get the result for the required product ID. Another assumption is that this service is running on port 10000. Just to make sure everything is running, a hit on a URL such as http://localhost:10000/product/1 should give some JSON as response. For the API Gateway, we will create another Spring Boot application with Zuul support. Zuul can be activated by just adding a simple @EnableZuulProxy annotation. The following is a simple code to start the simple Zuul proxy: @SpringBootApplication @EnableZuulProxy public class ApiGatewayExampleInSpring { public static void main(String[] args) { SpringApplication.run(ApiGatewayExampleInSpring.class, args); } }   Rest all the things are managed in configuration. In the application.properties file of the API Gateway, the content will be something as follows: zuul.routes.product.path=/product/** zuul.routes.produc.url=http://localhost:10000 ribbon.eureka.enabled=false server.port=8080  With this configuration, we are defining rules such as this: for any request for a URL such as /product/xxx, pass this request to http://localhost:10000. For outer world, it will be like http://localhost:8080/product/1, which will internally be transferred to the 10000 port. If we defined a spring.application.name variable as product in product detail microservice, then we don't need to define the URL path property here (zuul.routes.product.path=/product/** ), as Zuul, by default, will make it a URL/product. The example taken here for an API Gateway is not very intelligent, but this is a very capable API Gateway. Depending on the routes, filter, and caching defined in the Zuul's property, one can make a very powerful API Gateway. Summary In this article, you learned about the API Gateway, its need, and its pros and cons with the code example. Resources for Article:   Further resources on this subject: What are Microservices? [article] Microservices and Service Oriented Architecture [article] Breaking into Microservices Architecture [article]

This article by Justin M. Brant, the author of SolarWinds Server & Application Monitor: Deployment and Administration, covers enabling and configuring SNMP on Windows. (For more resources related to this topic, see here.) Procedures in this article are not required pre-deployment, as it is possible after deployment to populate SolarWinds SAM with nodes; however, it is recommended. Even after deployment, you should still enable and configure advanced monitoring services on your vital nodes. SolarWinds SAM uses three types of protocols to poll management data: Simple Network Management Protocol (SNMP): This is the most common network management service protocol. To utilize it, SNMP must be enabled and an SNMP community string must be assigned on the server, device, or application. The community string is essentially a password that is sent between a node and SolarWinds SAM. Once the community string is set and assigned, the node is permitted to expose management data to SolarWinds SAM, in the form of variables. Currently, there are three versions of SNMP: v1, v2c, and v3. SolarWinds SAM uses SNMPv2c by default. To poll using SNMPv1, you must disable SNMPv2c on the device. Similarly, to poll using SNMPv3, you must configure your devices and SolarWinds SAM accordingly. Windows Management Instrumentation (WMI): This has added functionality by incorporating Windows specifi c communications and security features. WMI comes preinstalled on Windows by default but is not automatically enabled and confi gured. WMI is not exclusive to Windows server platforms; it comes installed on all modern Microsoft operating systems, and can also be used to poll desktop operating systems, such as Windows 7. Internet Control Message Protocol (ICMP): This is the most basic of the three; it simply sends echo requests (pings) to a server or device for status, response time, and packet loss. SolarWinds SAM uses ICMP in conjunction with SNMP and WMI. Nodes can be confi gured to poll with ICMP exclusively, but you miss out on CPU, memory, and volume data. Some devices can only be polled with ICMP, although in most instances you will rarely use ICMP exclusively. Trying to decide between SNMP and WMI? SNMP is more standardized and provides data that you may not be able to poll with WMI, such as interface information. In addition, polling a single WMI-enabled node uses roughly five times the resources required to poll the same node with SNMP. This article will explain how to prepare for SolarWinds SAM deployment, by enabling and configuring network management services and protocols on: Windows servers. In this article we will reference service accounts. A service account is an account created to handoff credentials to SolarWinds SAM. Service accounts are a best practice primarily for security reasons, but also to ensure that user accounts do not become locked out. Enabling and configuring SNMP on Windows Procedures listed in this article will explain how to enable SNMP and then assign a community string, on Windows Server 2008 R2. All Windows server-related procedures in this book are performed on Windows Server 2008 R2. Procedures vary slightly in other supported versions. Installing an SNMP service on Windows This procedure explains how to install the SNMP service on Windows Server 2008 R2. Log in to a Windows server. Navigate to Start Menu | Control Panel | Administrative Tools | Server Manager. In order to see Administrative Tools in the Control Panel, you may need to select View by: Small Icons or Large Icons. Select Features and click on Add Features. Check SNMP Services, then click on Next and Install. Click on Close. Assigning an SNMP community string on Windows This procedure explains how to assign a community string on Windows 2008 R2, and ensure that the SNMP service is configured to run automatically on start up. Log in to a Windows server. Navigate to Start Menu | Control Panel | Administrative Tools | Services. Double-click on SNMP Service. On the General tab, select Automatic under Startup type. Select the Agent tab and ensure Physical, Applications, Internet, and End-to-end are all checked under the Service area. Optionally, enter a Contact person and system Location. Select the Security tab and click on the Add button under Accepted community names. Enter a Community Name and click on the Add button. For example, we used S4MS3rv3r. We recommend using something secure, as this is a password. Community String and Community Name mean the same thing.   READ ONLY community rights will normally suffice. A detailed explanation of community rights can be found on the author's blog: http://justinmbrant.blogspot.com/ Next, tick the Accept SNMP packets from these hosts radio button. Click on the Add button underneath the radio buttons and add the IP of the server you have designated as the SolarWinds SAM host. Once you complete these steps, the SNMP Service Properties Security tab should look something like the following screenshot. Notice that we used 192.168.1.3, as that is the IP of the server where we plan to deploy SolarWinds SAM. Summary In this article, we learned different types of protocols to poll management data. We also learned how to install SNMP as well as assign SNMP community string on Windows. Resources for Article: Further resources on this subject: The OpenFlow Controllers [Article] The GNS3 orchestra [Article] Network Monitoring Essentials [Article]

In this article, we will take a closer look at the Python libraries available for the QGIS Python developer, and also look at the various ways in which we can use  these libraries to perform useful tasks within QGIS. In particular, you will learn: How the QGIS Python libraries are based on the underlying C++ APIs How to use the C++ API documentation as a reference to work with the Python APIs How the PyQGIS libraries are organized The most important concepts and classes within the PyQGIS libraries and how to use them Some practical examples of performing useful tasks using PyQGIS About the QGIS Python APIs The QGIS system itself is written in C++ and has its own set of APIs that are also written in C++. The Python APIs are implemented as wrappers around these C++ APIs. For example, there is a Python class named QgisInterface that acts as a wrapper around a C++ class of the same name. All the methods, class variables, and the like, which are implemented by the C++ version of QgisInterface are made available through the Python wrapper. What this means is that when you access the Python QGIS APIs, you aren't accessing the API directly. Instead, the wrapper connects your code to the underlying C++ objects and methods, as follows :  Fortunately, in most cases, the QGIS Python wrappers simply hide away the complexity of the underlying C++ code, so the PyQGIS libraries work as you would expect them to. There are some gotchas, however, and we will cover these as they come up. Deciphering the C++ documentation As QGIS is implemented in C++, the documentation for the QGIS APIs is all based on C++. This can make it difficult for Python developers to understand and work with the QGIS APIs. For example, the API documentation for the QgsInterface.zoomToActiveLayer() method:  If you're not familiar with C++, this can be quite confusing. Fortunately, as a Python programmer, you can skip over much of this complexity because it doesn't apply to you. In particular: The virtual keyword is an implementation detail you don't need to worry about The word void indicates that the method doesn't return a value The double colons in QgisInterface::zoomToActiveLayer are simply a C++ convention for separating the class name from the method name Just like in Python, the parentheses show that the method doesn't take any parameters. So if you have an instance of QgisInterface (for example, as the standard iface variable available in the Python Console), you can call this method simply by typing the following: iface.zoomToActiveLayer() Now, let's take a look at a slightly more complex example: the C++ documentation for the QgisInterface.addVectorLayer() method looks like the following:  Notice how the virtual keyword is followed by QgsVectorLayer* instead of void. This is the return value for this method; it returns a QgsVector object. Technically speaking, * means that the method returns a pointer to an object of type QgsVectorLayer. Fortunately, Python wrappers automatically handle pointers, so you don't need to worry about this.  Notice the brief description at the bottom of the documentation for this method; while many of the C++ methods have very little, if any, additional information, other methods have quite extensive descriptions. Obviously, you should read these descriptions carefully as they tell you more about what the method does. Even without any description, the C++ documentation is still useful as it tells you what the method is called, what parameters it accepts, and what type of data is being returned. In the preceding method, you can see that there are three parameters listed in between the parentheses. As C++ is a strongly typed language, you have to define the type of each parameter when you define a function. This is helpful for Python programmers as it tells you what type of value to supply. Apart from QGIS objects, you might also encounter the following data types in the C++ documentation: Data type Description int A standard Python integer value long A standard Python long integer value float A standard Python floating point (real) number bool A Boolean value (true or false) QString A string value. Note that the QGIS Python wrappers automatically convert Python strings to C++ strings, so you don't need to deal with QString objects directly QList This object is used to encapsulate a list of other objects. For example, QList<QString*> represents a list of strings Just as in Python, a method can take default values for each parameter. For example, the QgisInterface.newProject() method looks like the following:  In this case, the thePromptToSaveFlag parameter has a default value, and this default value will be used if no value is supplied. In Python, classes are initialized using the __init__ method. In C++, this is called a constructor. For example, the constructor for the QgsLabel class looks like the following:  Just as in Python, C++ classes inherit the methods defined in their superclass. Fortunately, QGIS doesn't have an extensive class hierarchy, so most of the classes don't have a superclass. However, don't forget to check for a superclass if you can't find the method you're looking for in the documentation for the class itself. Finally, be aware that C++ supports the concept of method overloading. A single method can be defined more than once, where each version accepts a different set of parameters. For example, take a look at the constructor for the QgsRectangle class—you will see that there are four different versions of this method. The first version accepts the four coordinates as floating point numbers:  The second version constructs a rectangle using two QgsPoint objects:  The third version copies the coordinates from QRectF (which is a Qt data type) into a QgsRectangle object:  The final version copies the coordinates from another QgsRectangle object:  The C++ compiler chooses the correct method to use based on the parameters that have been supplied. Python has no concept of method overloading; just choose the version of the method that accepts the parameters you want to supply, and the QGIS Python wrappers will automatically choose the correct method for you. If you keep these guidelines in mind, deciphering the C++ documentation for QGIS isn't all that hard. It just looks more complicated than it really is, thanks to all the complexity specific to C++. However, it won't take long for your brain to start filtering out C++ and use the QGIS reference documentation almost as easily as if it was written for Python rather than C++. Organization of the QGIS Python libraries Now that we can understand the C++-oriented documentation, let's see how the PyQGIS libraries are structured. All of the PyQGIS libraries are organized under a package named qgis. You wouldn't normally import qgis directly, however, as all the interesting libraries are subpackages within this main package; here are the five packages that make up the PyQGIS library: qgis.core This provides access to the core GIS functionality used throughout QGIS. qgis.gui This defines a range of GUI widgets that you can include in your own programs. qgis.analysis This provides spatial analysis tools to analyze vector and raster format data. qgis.networkanalysis This provides tools to build and analyze topologies. qgis.utils This implements miscellaneous functions that allow you to work with the QGIS application using Python.  The first two packages (qgis.core and qgis.gui) implement the most important parts of the PyQGIS library, and it's worth spending some time to become more familiar with the concepts and classes they define. Now let's take a closer look at these two packages. The qgis.core package The qgis.core package defines fundamental classes used throughout the QGIS system. A large part of this package is dedicated to working with vector and raster format geospatial data, and displaying these types of data within a map. Let's take a look at how this is done. Maps and map layers A map consists of multiple layers drawn one on top of the other:  There are three types of map layers supported by QGIS: Vector layer: This layer draws geospatial features such as points, lines, and polygons Raster layer: This layer draws raster (bitmapped) data onto a map Plugin layer: This layer allows a plugin to draw directly onto a map Each of these types of map layers has a corresponding class within the qgis.core library. For example, a vector map layer will be represented by an object of type qgis.core.QgsVectorLayer. We will take a closer look at vector and raster map layers shortly. Before we do this, though, we need to learn how geospatial data (both vector and raster data) is positioned on a map. Coordinate reference systems Since the Earth is a three-dimensional object, maps will only represent the Earth's surface as a two-dimensional plane, so there has to be a way of translating points on the Earth's surface into (x,y) coordinates within a map. This is done using a Coordinate Reference System (CRS):  Globe image courtesy Wikimedia (http://commons.wikimedia.org/wiki/File:Rotating_globe.gif) A CRS has two parts: an ellipsoid, which is a mathematical model of the Earth's surface, and a projection, which is a formula that converts points on the surface of the spheroid into (x,y) coordinates on a map. Generally you won't need to worry about all these details. You can simply select the appropriate CRS that matches the CRS of the data you are using. However, as there are many different coordinate reference systems that have been devised over the years, it is vital that you use the correct CRS when plotting your geospatial data. If you don't do this, your features will be displayed in the wrong place or have the wrong shape. The majority of geospatial data available today uses the EPSG 4326 coordinate reference system (sometimes also referred to as WGS84). This CRS defines coordinates as latitude and longitude values. This is the default CRS used for new data imported into QGIS. However, if your data uses a different coordinate reference system, you will need to create and use a different CRS for your map layer. The qgis.core.QgsCoordinateReferenceSystem class represents a CRS. Once you create your coordinate reference system, you can tell your map layer to use that CRS when accessing the underlying data. For example: crs = QgsCoordinateReferenceSystem(4326,           QgsCoordinateReferenceSystem.EpsgCrsId))layer.setCrs(crs) Note that different map layers can use different coordinate reference systems. Each layer will use its CRS when drawing the contents of the layer onto the map. Vector layers A vector layer draws geospatial data onto a map in the form of points, lines, polygons, and so on. Vector-format geospatial data is typically loaded from a vector data source such as a shapefile or database. Other vector data sources can hold vector data in memory, or load data from a web service across the internet. A vector-format data source has a number of features, where each feature represents a single record within the data source. The qgis.core.QgsFeature class represents a feature within a data source. Each feature has the following principles: ID: This is the feature's unique identifier within the data source. Geometry: This is the underlying point, line, polygon, and so on, which represents the feature on the map. For example, a data source representing cities would have one feature for each city, and the geometry would typically be either a point that represents the center of the city, or a polygon (or a multipolygon) that represents the city's outline. Attributes: These are key/value pairs that provide additional information about the feature. For example, a city data source might have attributes such as total_area, population, elevation, and so on. Attribute values can be strings, integers, or floating point numbers. In QGIS, a data provider allows the vector layer to access the features within the data source. The data provider, an instance of qgis.core.QgsVectorDataProvider, includes: A geometry type that is stored in the data source A list of fields that provide information about the attributes stored for each feature The ability to search through the features within the data source, using the getFeatures() method and the QgsFeatureRequest class You can access the various vector (and also raster) data providers by using the qgis.core.QgsProviderRegistry class. The vector layer itself is represented by a qgis.core.QgsVectorLayer object. Each vector layer includes: Data provider: This is the connection to the underlying file or database that holds the geospatial information to be displayed Coordinate reference system: This indicates which CRS the geospatial data uses Renderer: This chooses how the features are to be displayed Let's take a closer look at the concept of a renderer and how features are displayed within a vector map layer. Displaying vector data The features within a vector map layer are displayed using a combination of renderer and symbol objects. The renderer chooses the symbol to use for a given feature, and the symbol that does the actual drawing. There are three basic types of symbols defined by QGIS: Marker symbol: This displays a point as a filled circle Line symbol: This draws a line using a given line width and color Fill symbol: This draws the interior of a polygon with a given color These three types of symbols are implemented as subclasses of the qgis.core.QgsSymbolV2 class: qgis.core.QgsMarkerSymbolV2 qgis.core.QgsLineSymbolV2 qgis.core.QgsFillSymbolV2 Internally, symbols are rather complex, as "symbol layers" allow multiple elements to be drawn on top of each other. In most cases, however, you can make use of the "simple" version of the symbol. This makes it easier to create a new symbol without having to deal with the internal complexity of symbol layers. For example: symbol = QgsMarkerSymbolV2.createSimple({'width' : 1.0, 'color' : "255,0,0"}) While symbols draw the features onto the map, a renderer is used to choose which symbol to use to draw a particular feature. In the simplest case, the same symbol is used for every feature within a layer. This is called a single symbol renderer, and is represented by the qgis.core.QgsSingleSymbolRenderV2 class. Other possibilities include: Categorized symbol renderer (qgis.core.QgsCategorizedSymbolRendererV2): This renderer chooses a symbol based on the value of an attribute. The categorized symbol renderer has a mapping from attribute values to symbols. Graduated symbol renderer (qgis.core.QgsGraduatedSymbolRendererV2): This type of renderer has a range of attribute, values, and maps each range to an appropriate symbol. Using a single symbol renderer is very straightforward: symbol = ...renderer = QgsSingleSymbolRendererV2(symbol)layer.setRendererV2(renderer) To use a categorized symbol renderer, you first define a list of qgis.core. QgsRendererCategoryV2 objects, and then use that to create the renderer. For example: symbol_male = ...symbol_female = ... categories = []categories.append(QgsRendererCategoryV2("M", symbol_male, "Male"))categories.append(QgsRendererCategoryV2("F", symbol_female, "Female"))renderer = QgsCategorizedSymbolRendererV2("", categories)renderer.setClassAttribute("GENDER")layer.setRendererV2(renderer) Notice that the QgsRendererCategoryV2 constructor takes three parameters: the desired value, the symbol used, and a label used to describe that category. Finally, to use a graduated symbol renderer, you define a list of qgis.core.QgsRendererRangeV2 objects and then use that to create your renderer. For example: symbol1 = ...symbol2 = ... ranges = []ranges.append(QgsRendererRangeV2(0, 10, symbol1, "Range 1"))ranges.append(QgsRendererRange(11, 20, symbol2, "Range 2")) renderer = QgsGraduatedSymbolRendererV2("", ranges)renderer.setClassAttribute("FIELD")layer.setRendererV2(renderer) Accessing vector data In addition to displaying the contents of a vector layer within a map, you can use Python to directly access the underlying data. This can be done using the data provider's getFeatures() method. For example, to iterate over all the features within the layer, you can do the following: provider = layer.dataProvider()for feature in provider.getFeatures(QgsFeatureRequest()):  ... If you want to search for features based on some criteria, you can use the QgsFeatureRequest object's setFilterExpression() method, as follows: provider = layer.dataProvider()request = QgsFeatureRequest()request.setFilterExpression('"GENDER" = "M"')for feature in provider.getFeatures(QgsFeatureRequest()):  ... Once you have the features, it's easy to get access to the feature's geometry, ID, and attributes. For example: geometry = feature.geometry()  id = feature.id()  name = feature.attribute("NAME") The object returned by the feature.geometry() call, which will be an instance of qgis.core.QgsGeometry, represents the feature's geometry. This object has a number of methods you can use to extract the underlying data and perform various geospatial calculations. Spatial indexes In the previous section, we searched for features based on their attribute values. There are times, though, when you might want to find features based on their position in space. For example, you might want to find all features that lie within a certain distance of a given point. To do this, you can use a spatial index, which indexes features according to their location and extent. Spatial indexes are represented in QGIS by the QgsSpatialIndex class. For performance reasons, a spatial index is not created automatically for each vector layer. However, it's easy to create one when you need it: provider = layer.dataProvider()index = QgsSpatialIndex()for feature in provider.getFeatures(QgsFeatureRequest()):  index.insertFeature(feature) Don't forget that you can use the QgsFeatureRequest.setFilterExpression() method to limit the set of features that get added to the index. Once you have the spatial index, you can use it to perform queries based on the position of the features. In particular: You can find one or more features that are closest to a given point using the nearestNeighbor() method. For example: features = index.nearestNeighbor(QgsPoint(long, lat), 5) Note that this method takes two parameters: the desired point as a QgsPoint object and the number of features to return. You can find all features that intersect with a given rectangular area by using the intersects() method, as follows: features = index.intersects(QgsRectangle(left, bottom, right, top)) Raster layers Raster-format geospatial data is essentially a bitmapped image, where each pixel or "cell" in the image corresponds to a particular part of the Earth's surface. Raster data is often organized into bands, where each band represents a different piece of information. A common use for bands is to store the red, green, and blue component of the pixel's color in a separate band. Bands might also represent other types of information, such as moisture level, elevation, or soil type. There are many ways in which raster information can be displayed. For example: If the raster data only has one band, the pixel value can be used as an index into a palette. The palette maps each pixel value maps to a particular color. If the raster data has only one band but no palette is provided. The pixel values can be used directly as a grayscale value; that is, larger numbers are lighter and smaller numbers are darker. Alternatively, the pixel values can be passed through a pseudocolor algorithm to calculate the color to be displayed. If the raster data has multiple bands, then typically, the bands would be combined to generate the desired color. For example, one band might represent the red component of the color, another band might represent the green component, and yet another band might represent the blue component. Alternatively, a multiband raster data source might be drawn using a palette, or as a grayscale or a pseudocolor image, by selecting a particular band to use for the color calculation. Let's take a closer look at how raster data can be drawn onto the map. Displaying raster data The drawing style associated with the raster band controls how the raster data will be displayed. The following drawing styles are currently supported: Drawing style Description PalettedColor For a single band raster data source, a palette maps each raster value to a color. SingleBandGray For a single band raster data source, the raster value is used directly as a grayscale value. SingleBandPseudoColor For a single band raster data source, the raster value is used to calculate a pseudocolor. PalettedSingleBandGray For a single band raster data source that has a palette, this drawing style tells QGIS to ignore the palette and use the raster value directly as a grayscale value. PalettedSingleBandPseudoColor For a single band raster data source that has a palette, this drawing style tells QGIS to ignore the palette and use the raster value to calculate a pseudocolor. MultiBandColor For multiband raster data sources, use a separate band for each of the red, green, and blue color components. For this drawing style, the setRedBand(), setGreenBand(), and setBlueBand() methods can be used to choose which band to use for each color component. MultiBandSingleBandGray For multiband raster data sources, choose a single band to use as the grayscale color value. For this drawing style, use the setGrayBand() method to specify the band to use. MultiBandSingleBandPseudoColor For multiband raster data sources, choose a single band to use to calculate a pseudocolor. For this drawing style, use the setGrayBand() method to specify the band to use.  To set the drawing style, use the layer.setDrawingStyle() method, passing in a string that contains the name of the desired drawing style. You will also need to call the various setXXXBand() methods, as described in the preceding table, to tell the raster layer which bands contain the value(s) to use to draw each pixel. Note that QGIS doesn't automatically update the map when you call the preceding functions to change the way the raster data is displayed. To have your changes displayed right away, you'll need to do the following: Turn off raster image caching. This can be done by calling layer.setImageCache(None). Tell the raster layer to redraw itself, by calling layer.triggerRepaint(). Accessing raster data As with vector-format data, you can access the underlying raster data via the data provider's identify() method. The easiest way to do this is to pass in a single coordinate and retrieve the value or values of that coordinate. For example: provider = layer.dataProvider()values = provider.identify(QgsPoint(x, y),              QgsRaster.IdentifyFormatValue)if values.isValid():  for band,value in values.results().items():    ... As you can see, you need to check whether the given coordinate exists within the raster data (using the isValid() call). The values.results() method returns a dictionary that maps band numbers to values. Using this technique, you can extract all the underlying data associated with a given coordinate within the raster layer. You can also use the provider.block() method to retrieve the band data for a large number of coordinates all at once. We will look at how to do this later in this article. Other useful qgis.core classes Apart from all the classes and functionality involved in working with data sources and map layers, the qgis.core library also defines a number of other classes that you might find useful: Class Description QgsProject This represents the current QGIS project. Note that this is a singleton object, as only one project can be open at a time. The QgsProject class is responsible for loading and storing properties, which can be useful for plugins. QGis This class defines various constants, data types, and functions used throughout the QGIS system. QgsPoint This is a generic class that stores the coordinates for a point within a two-dimensional plane. QgsRectangle This is a generic class that stores the coordinates for a rectangular area within a two-dimensional plane. QgsRasterInterface This is the base class to use for processing raster data. This can be used, to reproject a set of raster data into a new coordinate system, to apply filters to change the brightness or color of your raster data, to resample the raster data, and to generate new raster data by rendering the existing data in various ways. QgsDistanceArea This class can be used to calculate distances and areas for a given geometry, automatically converting from the source coordinate reference system into meters. QgsMapLayerRegistry This class provides access to all the registered map layers in the current project. QgsMessageLog This class provides general logging features within a QGIS program. This lets you send debugging messages, warnings, and errors to the QGIS "Log Messages" panel.  The qgis.gui package The qgis.gui package defines a number of user-interface widgets that you can include in your programs. Let's start by looking at the most important qgis.gui classes, and follow this up with a brief look at some of the other classes that you might find useful. The QgisInterface class QgisInterface represents the QGIS system's user interface. It allows programmatic access to the map canvas, the menu bar, and other parts of the QGIS application. When running Python code within a script or a plugin, or directly from the QGIS Python console, a reference to QgisInterface is typically available through the iface global variable. The QgisInterface object is only available when running the QGIS application itself. If you are running an external application and import the PyQGIS library into your application, QgisInterface won't be available. Some of the more important things you can do with the QgisInterface object are: Get a reference to the list of layers within the current QGIS project via the legendInterface() method. Get a reference to the map canvas displayed within the main application window, using the mapCanvas() method. Retrieve the currently active layer within the project, using the activeLayer() method, and set the currently active layer by using the setActiveLayer() method. Get a reference to the application's main window by calling the mainWindow() method. This can be useful if you want to create additional Qt windows or dialogs that use the main window as their parent. Get a reference to the QGIS system's message bar by calling the messageBar() method. This allows you to display messages to the user directly within the QGIS main window. The QgsMapCanvas class The map canvas is responsible for drawing the various map layers into a window. The QgsMapCanvas class represents a map canvas. This class includes: A list of the currently shown map layers. This can be accessed using the layers() method. Note that there is a subtle difference between the list of map layers available within the map canvas and the list of map layers included in the QgisInterface.legendInterface() method. The map canvas's list of layers only includes the list of layers currently visible, while QgisInterface.legendInterface() returns all the map layers, including those that are currently hidden.  The map units used by this map (meters, feet, degrees, and so on). The map's units can be retrieved by calling the mapUnits() method. An extent,which is the area of the map that is currently shown within the canvas. The map's extent will change as the user zooms in and out, and pans across the map. The current map extent can be obtained by calling the extent() method. A current map tool that controls the user's interaction with the contents of the map canvas. The current map tool can be set using the setMapTool() method, and you can retrieve the current map tool (if any) by calling the mapTool() method. A background color used to draw the background behind all the map layers. You can change the map's background color by calling the canvasColor() method. A coordinate transform that converts from map coordinates (that is, coordinates in the data source's coordinate reference system) to pixels within the window. You can retrieve the current coordinate transform by calling the getCoordinateTransform() method. The QgsMapCanvasItem class A map canvas item is an item drawn on top of the map canvas. The map canvas item will appear in front of the map layers. While you can create your own subclass of QgsMapCanvasItem if you want to draw custom items on top of the map canvas, it would be more useful for you to make use of an existing subclass that will do the work for you. There are currently three subclasses of QgsMapCanvasItem that you might find useful: QgsVertexMarker: This draws an icon (an "X", a "+", or a small box) centered around a given point on the map. QgsRubberBand: This draws an arbitrary polygon or polyline onto the map. It is intended to provide visual feedback as the user draws a polygon onto the map. QgsAnnotationItem: This is used to display additional information about a feature, in the form of a balloon that is connected to the feature. The QgsAnnotationItem class has various subclasses that allow you to customize the way the information is displayed. The QgsMapTool class A map tool allows the user to interact with and manipulate the map canvas, capturing mouse events and responding appropriately. A number of QgsMapTool subclasses provide standard map interaction behavior such as clicking to zoom in, dragging to pan the map, and clicking on a feature to identify it. You can also create your own custom map tools by subclassing QgsMapTool and implementing the various methods that respond to user-interface events such as pressing down the mouse button, dragging the canvas, and so on. Once you have created a map tool, you can allow the user to activate it by associating the map tool with a toolbar button. Alternatively, you can activate it from within your Python code by calling the mapCanvas.setMapTool(...) method. We will look at the process of creating your own custom map tool in the section Using the PyQGIS library in the following table: Other useful qgis.gui classes While the qgis.gui package defines a large number of classes, the ones you are most likely to find useful are given in the following table: Class Description QgsLegendInterface This provides access to the map legend, that is, the list of map layers within the current project. Note that map layers can be grouped, hidden, and shown within the map legend. QgsMapTip This displays a tip on a map canvas when the user holds the mouse over a feature. The map tip will show the display field for the feature; you can set this by calling layer.setDisplayField("FIELD"). QgsColorDialog This is a dialog box that allows the user to select a color. QgsDialog This is a generic dialog with a vertical box layout and a button box, making it easy to add content and standard buttons to your dialog. QgsMessageBar This is a user interface widget for displaying non-blocking messages to the user. We looked at the message bar class in the previous article. QgsMessageViewer This is a generic class that displays long messages to the user within a modal dialog. QgsBlendModeComboBox QgsBrushStyleComboBox QgsColorRampComboBox QgsPenCapStyleComboBox QgsPenJoinStyleComboBox QgsScaleComboBox These QComboBox user-interface widgets allow you to prompt the user for various drawing options. With the exception of the QgsScaleComboBox, which lets the user choose a map scale, all the other QComboBox subclasses let the user choose various Qt drawing options.  Using the PyQGIS library In the previous section, we looked at a number of classes provided by the PyQGIS library. Let's make use of these classes to perform some real-world geospatial development tasks. Analyzing raster data We're going to start by writing a program to load in some raster-format data and analyze its contents. To make this more interesting, we'll use a Digital Elevation Model (DEM) file, which is a raster format data file that contains elevation data. The Global Land One-Kilometer Base Elevation Project (GLOBE) provides free DEM data for the world, where each pixel represents one square kilometer of the Earth's surface. GLOBE data can be downloaded from http://www.ngdc.noaa.gov/mgg/topo/gltiles.html. Download the E tile, which includes the western half of the USA. The resulting file, which is named e10g, contains the height information you need. You'll also need to download the e10g.hdr header file so that QGIS can read the file—you can download this from http://www.ngdc.noaa.gov/mgg/topo/elev/esri/hdr. Once you've downloaded these two files, put them together into a convenient directory. You can now load the DEM data into QGIS using the following code: registry = QgsProviderRegistry.instance()provider = registry.provider("gdal", "/path/to/e10g") Unfortunately, there is a slight complexity here. Since QGIS doesn't know which coordinate reference system is used for the data, it displays a dialog box that asks you to choose the CRS. Since the GLOBE DEM data is in the WGS84 CRS, which QGIS uses by default, this dialog box is redundant. To disable it, you need to add the following to the top of your program: from PyQt4.QtCore import QSettingsQSettings().setValue("/Projections/defaultBehaviour", "useGlobal") Now that we've loaded our raster DEM data into QGIS, we can analyze. There are lots of things we can do with DEM data, so let's calculate how often each unique elevation value occurs within the data. Notice that we're loading the DEM data directly using QgsRasterDataProvider. We don't want to display this information on a map, so we don't want (or need) to load it into QgsRasterLayer.  Since the DEM data is in a raster format, you need to iterate over the individual pixels or cells to get each height value. The provider.xSize() and provider.ySize() methods tell us how many cells are in the DEM, while the provider.extent() method gives us the area of the Earth's surface covered by the DEM. Using this information, we can extract the individual elevation values from the contents of the DEM in the following way: raster_extent = provider.extent()raster_width = provider.xSize()raster_height = provider.ySize()block = provider.block(1, raster_extent, raster_width, raster_height) The returned block variable is an object of type QgsRasterBlock, which is essentially a two-dimensional array of values. Let's iterate over the raster and extract the individual elevation values: for x in range(raster_width):  for y in range(raster_height):    elevation = block.value(x, y)    .... Now that we've loaded the individual elevation values, it's easy to build a histogram out of those values. Here is the entire program to load the DEM data into memory, and calculate, and display the histogram: from PyQt4.QtCore import QSettingsQSettings().setValue("/Projections/defaultBehaviour", "useGlobal")registry = QgsProviderRegistry.instance()provider = registry.provider("gdal", "/path/to/e10g") raster_extent = provider.extent()raster_width = provider.xSize()raster_height = provider.ySize()no_data_value = provider.srcNoDataValue(1) histogram = {} # Maps elevation to number of occurrences. block = provider.block(1, raster_extent, raster_width,            raster_height)if block.isValid():  for x in range(raster_width):    for y in range(raster_height):      elevation = block.value(x, y)      if elevation != no_data_value:        try:          histogram[elevation] += 1        except KeyError:          histogram[elevation] = 1 for height in sorted(histogram.keys()):  print height, histogram[height] Note that we've added a no data value check to the code. Raster data often includes pixels that have no value associated with them. In the case of a DEM, elevation data is only provided for areas of land; pixels over the sea have no elevation, and we have to exclude them, or our histogram will be inaccurate. Manipulating vector data and saving it to a shapefile Let's create a program that takes two vector data sources, subtracts one set of vectors from the other, and saves the resulting geometries into a new shapefile. Along the way, we'll learn a few important things about the PyQGIS library. We'll be making use of the QgsGeometry.difference() function. This function performs a geometrical subtraction of one geometry from another, similar to this:  Let's start by asking the user to select the first shapefile and open up a vector data provider for that file: filename_1 = QFileDialog.getOpenFileName(iface.mainWindow(),                     "First Shapefile",                     "~", "*.shp")if not filename_1:  return registry = QgsProviderRegistry.instance()provider_1 = registry.provider("ogr", filename_1) We can then read the geometries from that file into memory: geometries_1 = []for feature in provider_1.getFeatures(QgsFeatureRequest()):  geometries_1.append(QgsGeometry(feature.geometry())) This last line of code does something very important that may not be obvious at first. Notice that we use the following: QgsGeometry(feature.geometry()) We use the preceding line instead of the following: feature.geometry() This creates a new instance of the QgsGeometry object, copying the geometry into a new object, rather than just adding the existing geometry object to the list. We have to do this because of a limitation of the way the QGIS Python wrappers work: the feature.geometry() method returns a reference to the geometry, but the C++ code doesn't know that you are storing this reference away in your Python code. So, when the feature is no longer needed, the memory used by the feature's geometry is also released. If you then try to access that geometry later on, the entire QGIS system will crash. To get around this, we make a copy of the geometry so that we can refer to it even after the feature's memory has been released. Now that we've loaded our first set of geometries into memory, let's do the same for the second shapefile: filename_2 = QFileDialog.getOpenFileName(iface.mainWindow(),                     "Second Shapefile",                     "~", "*.shp")if not filename_2:  return provider_2 = registry.provider("ogr", filename_2) geometries_2 = []for feature in provider_2.getFeatures(QgsFeatureRequest()):  geometries_2.append(QgsGeometry(feature.geometry())) With the two sets of geometries loaded into memory, we're ready to start subtracting one from the other. However, to make this process more efficient, we will combine the geometries from the second shapefile into one large geometry, which we can then subtract all at once, rather than subtracting one at a time. This will make the subtraction process much faster: combined_geometry = Nonefor geometry in geometries_2:  if combined_geometry == None:    combined_geometry = geometry  else:    combined_geometry = combined_geometry.combine(geometry) We can now calculate the new set of geometries by subtracting one from the other: dst_geometries = []for geometry in geometries_1:  dst_geometry = geometry.difference(combined_geometry)  if not dst_geometry.isGeosValid(): continue  if dst_geometry.isGeosEmpty(): continue  dst_geometries.append(dst_geometry) Notice that we check to ensure that the destination geometry is mathematically valid and isn't empty. Invalid geometries are a common problem when manipulating complex shapes. There are options for fixing them, such as splitting apart multi-geometries and performing a buffer operation.  Our last task is to save the resulting geometries into a new shapefile. We'll first ask the user for the name of the destination shapefile: dst_filename = QFileDialog.getSaveFileName(iface.mainWindow(),                      "Save results to:",                      "~", "*.shp")if not dst_filename:  return We'll make use of a vector file writer to save the geometries into a shapefile. Let's start by initializing the file writer object: fields = QgsFields()writer = QgsVectorFileWriter(dst_filename, "ASCII", fields,               dst_geometries[0].wkbType(),               None, "ESRI Shapefile")if writer.hasError() != QgsVectorFileWriter.NoError:  print "Error!"  return We don't have any attributes in our shapefile, so the fields list is empty. Now that the writer has been set up, we can save the geometries into the file: for geometry in dst_geometries:  feature = QgsFeature()  feature.setGeometry(geometry)  writer.addFeature(feature) Now that all the data has been written to the disk, let's display a message box that informs the user that we've finished: QMessageBox.information(iface.mainWindow(), "",            "Subtracted features saved to disk.") As you can see, creating a new shapefile is very straightforward in PyQGIS, and it's easy to manipulate geometries using Python—just so long as you copy QgsGeometry you want to keep around. If your Python code starts to crash while manipulating geometries, this is probably the first thing you should look for. Using different symbols for different features within a map Let's use World Borders Dataset that you downloaded in the previous article to draw a world map, using different symbols for different continents. This is a good example of using a categorized symbol renderer, though we'll combine it into a script that loads the shapefile into a map layer, and sets up the symbols and map renderer to display the map exactly as you want it. We'll then save this map as an image. Let's start by creating a map layer to display the contents of the World Borders Dataset shapefile: layer = iface.addVectorLayer("/path/to/TM_WORLD_BORDERS-0.3.shp",               "continents", "ogr") Each unique region code in the World Borders Dataset shapefile corresponds to a continent. We want to define the name and color to use for each of these regions, and use this information to set up the various categories to use when displaying the map: from PyQt4.QtGui import QColorcategories = []for value,color,label in [(0,   "#660000", "Antarctica"),                          (2,   "#006600", "Africa"),                          (9,   "#000066", "Oceania"),                          (19,  "#660066", "The Americas"),                          (142, "#666600", "Asia"),                          (150, "#006666", "Europe")]:  symbol = QgsSymbolV2.defaultSymbol(layer.geometryType())  symbol.setColor(QColor(color))  categories.append(QgsRendererCategoryV2(value, symbol, label)) With these categories set up, we simply update the map layer to use a categorized renderer based on the value of the region attribute, and then redraw the map: layer.setRendererV2(QgsCategorizedSymbolRendererV2("region",                          categories))layer.triggerRepaint() There's only one more thing to do; since this is a script that can be run multiple times, let's have our script automatically remove the existing continents layer, if it exists, before adding a new one. To do this, we can add the following to the start of our script: layer_registry = QgsMapLayerRegistry.instance() for layer in layer_registry.mapLayersByName("continents"):   layer_registry.removeMapLayer(layer.id()) When our script is running, it will create one (and only one) layer that shows the various continents in different colors. These will appear as different shades of gray in the printed article, but the colors will be visible on the computer screen: Now, let's use the same data set to color each country based on its relative population. We'll start by removing the existing population layer, if it exists: layer_registry = QgsMapLayerRegistry.instance()for layer in layer_registry.mapLayersByName("population"):  layer_registry.removeMapLayer(layer.id()) Next, we open the World Borders Dataset into a new layer called "population": layer = iface.addVectorLayer("/path/to/TM_WORLD_BORDERS-0.3.shp",               "population", "ogr") We then need to set up our various population ranges: from PyQt4.QtGui import QColorranges = []for min_pop,max_pop,color in [(0,        99999,     "#332828"),                              (100000,   999999,    "#4c3535"),                              (1000000,  4999999,   "#663d3d"),                              (5000000,  9999999,   "#804040"),                              (10000000, 19999999,  "#993d3d"),                              (20000000, 49999999,  "#b33535"),                              (50000000, 999999999, "#cc2828")]:  symbol = QgsSymbolV2.defaultSymbol(layer.geometryType())  symbol.setColor(QColor(color))  ranges.append(QgsRendererRangeV2(min_pop, max_pop,                   symbol, "")) Now that we have our population ranges and their associated colors, we simply set up a graduated symbol renderer to choose a symbol based on the value of the pop2005 attribute, and tell the map to redraw itself: layer.setRendererV2(QgsGraduatedSymbolRendererV2("pop2005",                         ranges))layer.triggerRepaint() The result will be a map layer that shades each country according to its population:  Calculating the distance between two user-defined points  In our final example of using the PyQGIS library, we'll write some code that, when run, starts listening for mouse events from the user. If the user clicks on a point, drags the mouse, and then releases the mouse button again, we will display the distance between those two points. This is a good example of how to add your  own map interaction logic to QGIS, using the QgsMapTool class. This is the basic structure for our QgsMapTool subclass: class DistanceCalculator(QgsMapTool):  def __init__(self, iface):    QgsMapTool.__init__(self, iface.mapCanvas())    self.iface = iface   def canvasPressEvent(self, event):    ...   def canvasReleaseEvent(self, event):    ... To make this map tool active, we'll create a new instance of it and pass it to the mapCanvas.setMapTool() method. Once this is done, our canvasPressEvent() and canvasReleaseEvent() methods will be called whenever the user clicks or releases the mouse button over the map canvas. Let's start with the code that handles the user clicking on the canvas. In this method, we're going to convert from the pixel coordinates that the user clicked on to the map coordinates (that is, a latitude and longitude value). We'll then remember these coordinates so that we can refer to them later. Here is the necessary code: def canvasPressEvent(self, event):  transform = self.iface.mapCanvas().getCoordinateTransform()  self._startPt = transform.toMapCoordinates(event.pos().x(),                        event.pos().y()) When the canvasReleaseEvent() method is called, we'll want to do the same with the point at which the user released the mouse button: def canvasReleaseEvent(self, event):  transform = self.iface.mapCanvas().getCoordinateTransform()  endPt = transform.toMapCoordinates(event.pos().x(),                    event.pos().y()) Now that we have the two desired coordinates, we'll want to calculate the distance between them. We can do this using a QgsDistanceArea object:      crs = self.iface.mapCanvas().mapRenderer().destinationCrs()  distance_calc = QgsDistanceArea()  distance_calc.setSourceCrs(crs)  distance_calc.setEllipsoid(crs.ellipsoidAcronym())  distance_calc.setEllipsoidalMode(crs.geographicFlag())  distance = distance_calc.measureLine([self._startPt,                     endPt]) / 1000 Notice that we divide the resulting value by 1000. This is because the QgsDistanceArea object returns the distance in meters, and we want to display the distance in kilometers. Finally, we'll display the calculated distance in the QGIS message bar:   messageBar = self.iface.messageBar()  messageBar.pushMessage("Distance = %d km" % distance,              level=QgsMessageBar.INFO,              duration=2) Now that we've created our map tool, we need to activate it. We can do this by adding the following to the end of our script: calculator = DistanceCalculator(iface)iface.mapCanvas().setMapTool(calculator) With the map tool activated, the user can click and drag on the map. When the mouse button is released, the distance (in kilometers) between the two points will be displayed in the message bar: Summary In this article, we took an in-depth look at the PyQGIS libraries and how you can use them in your own programs. We learned that the QGIS Python libraries are implemented as wrappers around the QGIS APIs implemented in C++. We saw how Python programmers can understand and work with the QGIS reference documentation, even though it is written for C++ developers. We also looked at the way the PyQGIS libraries are organized into different packages, and learned about the most important classes defined in the qgis.core and qgis.gui packages. We then saw how a coordinate reference systems (CRS) is used to translate from points on the three-dimensional surface of the Earth to coordinates within a two-dimensional map plane. We learned that vector format data is made up of features, where each feature has an ID, a geometry, and a set of attributes, and that symbols are used to draw vector geometries onto a map layer, while renderers are used to choose which symbol to use for a given feature. We learned how a spatial index can be used to speed up access to vector features. Next, we saw how raster format data is organized into bands that represent information such as color, elevation, and so on, and looked at the various ways in which a raster data source can be displayed within a map layer. Along the way, we learned how to access the contents of a raster data source. Finally, we looked at various techniques for performing useful tasks using the PyQGIS library. In the next article, we will learn more about QGIS Python plugins, and then go on to use the plugin architecture as a way of implementing a useful feature within a mapping application. Resources for Article:   Further resources on this subject: QGIS Feature Selection Tools [article] Server Logs [article]

In this article by Silas Toms, author of the book ArcPy and ArcGIS – Geospatial Analysis with Python we will see how programming languages share a concept that has aided programmers for decades: functions. The idea of a function, loosely speaking, is to create blocks of code that will perform an action on a piece of data, transforming it as required by the programmer and returning the transformed data back to the main body of code. Functions are used because they solve many different needs within programming. Functions reduce the need to write repetitive code, which in turn reduces the time needed to create a script. They can be used to create ranges of numbers (the range() function), or to determine the maximum value of a list (the max function), or to create a SQL statement to select a set of rows from a feature class. They can even be copied and used in another script or included as part of a module that can be imported into scripts. Function reuse has the added bonus of making programming more useful and less of a chore. When a scripter starts writing functions, it is a major step towards making programming part of a GIS workflow. (For more resources related to this topic, see here.) Technical definition of functions Functions, also called subroutines or procedures in other programming languages, are blocks of code that have been designed to either accept input data and transform it, or provide data to the main program when called without any input required. In theory, functions will only transform data that has been provided to the function as a parameter; it should not change any other part of the script that has not been included in the function. To make this possible, the concept of namespaces is invoked. Namespaces make it possible to use a variable name within a function, and allow it to represent a value, while also using the same variable name in another part of the script. This becomes especially important when importing modules from other programmers; within that module and its functions, the variables that it contains might have a variable name that is the same as a variable name within the main script. In a high-level programming language such as Python, there is built-in support for functions, including the ability to define function names and the data inputs (also known as parameters). Functions are created using the keyword def plus a function name, along with parentheses that may or may not contain parameters. Parameters can also be defined with default values, so parameters only need to be passed to the function when they differ from the default. The values that are returned from the function are also easily defined. A first function Let's create a function to get a feel for what is possible when writing functions. First, we need to invoke the function by providing the def keyword and providing a name along with the parentheses. The firstFunction() will return a string when called: def firstFunction():    'a simple function returning a string'    return "My First Function" >>>firstFunction() The output is as follows: 'My First Function' Notice that this function has a documentation string or doc string (a simple function returning a string) that describes what the function does; this string can be called later to find out what the function does, using the __doc__ internal function: >>>print firstFunction.__doc__ The output is as follows: 'a simple function returning a string' The function is defined and given a name, and then the parentheses are added followed by a colon. The following lines must then be indented (a good IDE will add the indention automatically). The function does not have any parameters, so the parentheses are empty. The function then uses the keyword return to return a value, in this case a string, from the function. Next, the function is called by adding parentheses to the function name. When it is called, it will do what it has been instructed to do: return the string. Functions with parameters Now let's create a function that accepts parameters and transforms them as needed. This function will accept a number and multiply it by 3: def secondFunction(number):    'this function multiples numbers by 3'    return number *3 >>> secondFunction(4) The output is as follows: 12 The function has one flaw, however; there is no assurance that the value passed to the function is a number. We need to add a conditional to the function to make sure it does not throw an exception: def secondFunction(number):    'this function multiples numbers by 3'    if type(number) == type(1) or type(number) == type(1.0):        return number *3 >>> secondFunction(4.0) The output is as follows: 12.0 >>>secondFunction(4) The output is as follows: 12 >>>secondFunction("String") >>> The function now accepts a parameter, checks what type of data it is, and returns a multiple of the parameter whether it is an integer or a function. If it is a string or some other data type, as shown in the last example, no value is returned. There is one more adjustment to the simple function that we should discuss: parameter defaults. By including default values in the definition of the function, we avoid having to provide parameters that rarely change. If, for instance, we wanted a different multiplier than 3 in the simple function, we would define it like this: def thirdFunction(number, multiplier=3):    'this function multiples numbers by 3'    if type(number) == type(1) or type(number) == type(1.0):        return number *multiplier >>>thirdFunction(4) The output is as follows: 12 >>>thirdFunction(4,5) The output is as follows: 20 The function will work when only the number to be multiplied is supplied, as the multiplier has a default value of 3. However, if we need another multiplier, the value can be adjusted by adding another value when calling the function. Note that the second value doesn't have to be a number as there is no type checking on it. Also, the default value(s) in a function must follow the parameters with no defaults (or all parameters can have a default value and the parameters can be supplied to the function in order or by name). Using functions to replace repetitive code One of the main uses of functions is to ensure that the same code does not have to be written over and over. The first portion of the script that we could convert into a function is the three ArcPy functions. Doing so will allow the script to be applicable to any of the stops in the Bus Stop feature class and have an adjustable buffer distance: bufferDist = 400 buffDistUnit = "Feet" lineName = '71 IB' busSignage = 'Ferry Plaza' sqlStatement = "NAME = '{0}' AND BUS_SIGNAG = '{1}'" def selectBufferIntersect(selectIn,selectOut,bufferOut,     intersectIn, intersectOut, sqlStatement,   bufferDist, buffDistUnit, lineName, busSignage):    'a function to perform a bus stop analysis'    arcpy.Select_analysis(selectIn, selectOut, sqlStatement.format(lineName, busSignage))    arcpy.Buffer_analysis(selectOut, bufferOut, "{0} {1}".format(bufferDist), "FULL", "ROUND", "NONE", "")    arcpy.Intersect_analysis("{0} #;{1} #".format(bufferOut, intersectIn), intersectOut, "ALL", "", "INPUT")    return intersectOut This function demonstrates how the analysis can be adjusted to accept the input and output feature class variables as parameters, along with some new variables. The function adds a variable to replace the SQL statement and variables to adjust the bus stop, and also tweaks the buffer distance statement so that both the distance and the unit can be adjusted. The feature class name variables, defined earlier in the script, have all been replaced with local variable names; while the global variable names could have been retained, it reduces the portability of the function. The next function will accept the result of the selectBufferIntersect() function and search it using the Search Cursor, passing the results into a dictionary. The dictionary will then be returned from the function for later use: def createResultDic(resultFC):    'search results of analysis and create results dictionary' dataDictionary = {}      with arcpy.da.SearchCursor(resultFC, ["STOPID","POP10"]) as cursor:        for row in cursor:            busStopID = row[0]            pop10 = row[1]            if busStopID not in dataDictionary.keys():                dataDictionary[busStopID] = [pop10]            else:                dataDictionary[busStopID].append(pop10)    return dataDictionary This function only requires one parameter: the feature class returned from the searchBufferIntersect() function. The results holding dictionary is first created, then populated by the search cursor, with the busStopid attribute used as a key, and the census block population attribute added to a list assigned to the key. The dictionary, having been populated with sorted data, is returned from the function for use in the final function, createCSV(). This function accepts the dictionary and the name of the output CSV file as a string: def createCSV(dictionary, csvname): 'a function takes a dictionary and creates a CSV file'    with open(csvname, 'wb') as csvfile:        csvwriter = csv.writer(csvfile, delimiter=',')        for busStopID in dictionary.keys():            popList = dictionary[busStopID]            averagePop = sum(popList)/len(popList)            data = [busStopID, averagePop]            csvwriter.writerow(data) The final function creates the CSV using the csv module. The name of the file, a string, is now a customizable parameter (meaning the script name can be any valid file path and text file with the extension .csv). The csvfile parameter is passed to the CSV module's writer method and assigned to the variable csvwriter, and the dictionary is accessed and processed, and passed as a list to csvwriter to be written to the CSV file. The csv.writer() method processes each item in the list into the CSV format and saves the final result. Open the CSV file with Excel or a text editor such as Notepad. To run the functions, we will call them in the script following the function definitions: analysisResult = selectBufferIntersect(Bus_Stops,Inbound71, Inbound71_400ft_buffer, CensusBlocks2010, Intersect71Census, bufferDist, lineName,                busSignage ) dictionary = createResultDic(analysisResult) createCSV(dictionary,r'C:\Projects\Output\Averages.csv') Now, the script has been divided into three functions, which replace the code of the first modified script. The modified script looks like this: # -*- coding: utf-8 -*- # --------------------------------------------------------------------------- # 8662_Chapter4Modified1.py # Created on: 2014-04-22 21:59:31.00000 #   (generated by ArcGIS/ModelBuilder) # Description: # Adjusted by Silas Toms # 2014 05 05 # ---------------------------------------------------------------------------   # Import arcpy module import arcpy import csv   # Local variables: Bus_Stops = r"C:\Projects\PacktDB.gdb\SanFrancisco\Bus_Stops" CensusBlocks2010 = r"C:\Projects\PacktDB.gdb\SanFrancisco\CensusBlocks2010" Inbound71 = r"C:\Projects\PacktDB.gdb\Chapter3Results\Inbound71" Inbound71_400ft_buffer = r"C:\Projects\PacktDB.gdb\Chapter3Results\Inbound71_400ft_buffer" Intersect71Census = r"C:\Projects\PacktDB.gdb\Chapter3Results\Intersect71Census" bufferDist = 400 lineName = '71 IB' busSignage = 'Ferry Plaza' def selectBufferIntersect(selectIn,selectOut,bufferOut,intersectIn,                          intersectOut, bufferDist,lineName, busSignage ):    arcpy.Select_analysis(selectIn,                          selectOut,                           "NAME = '{0}' AND BUS_SIGNAG = '{1}'".format(lineName, busSignage))    arcpy.Buffer_analysis(selectOut,                          bufferOut,                          "{0} Feet".format(bufferDist),                          "FULL", "ROUND", "NONE", "")    arcpy.Intersect_analysis("{0} #;{1} #".format(bufferOut,intersectIn),                              intersectOut, "ALL", "", "INPUT")    return intersectOut   def createResultDic(resultFC):    dataDictionary = {}       with arcpy.da.SearchCursor(resultFC,                                ["STOPID","POP10"]) as cursor:        for row in cursor:            busStopID = row[0]            pop10 = row[1]            if busStopID not in dataDictionary.keys():                dataDictionary[busStopID] = [pop10]            else:                dataDictionary[busStopID].append(pop10)    return dataDictionary   def createCSV(dictionary, csvname):    with open(csvname, 'wb') as csvfile:        csvwriter = csv.writer(csvfile, delimiter=',')        for busStopID in dictionary.keys():            popList = dictionary[busStopID]            averagePop = sum(popList)/len(popList)            data = [busStopID, averagePop]            csvwriter.writerow(data) analysisResult = selectBufferIntersect(Bus_Stops,Inbound71, Inbound71_400ft_buffer,CensusBlocks2010,Intersect71Census, bufferDist,lineName, busSignage ) dictionary = createResultDic(analysisResult) createCSV(dictionary,r'C:\Projects\Output\Averages.csv') print "Data Analysis Complete" Further generalization of the functions, while we have created functions from the original script that can be used to extract more data about bus stops in San Francisco, our new functions are still very specific to the dataset and analysis for which they were created. This can be very useful for long and laborious analysis for which creating reusable functions is not necessary. The first use of functions is to get rid of the need to repeat code. The next goal is to then make that code reusable. Let's discuss some ways in which we can convert the functions from one-offs into reusable functions or even modules. First, let's examine the first function: def selectBufferIntersect(selectIn,selectOut,bufferOut,intersectIn,                          intersectOut, bufferDist,lineName, busSignage ):    arcpy.Select_analysis(selectIn,                          selectOut,                          "NAME = '{0}' AND BUS_SIGNAG = '{1}'".format(lineName, busSignage))    arcpy.Buffer_analysis(selectOut,                          bufferOut,                          "{0} Feet".format(bufferDist),                          "FULL", "ROUND", "NONE", "")    arcpy.Intersect_analysis("{0} #;{1} #".format(bufferOut,intersectIn),                              intersectOut, "ALL", "", "INPUT")    return intersectOut This function appears to be pretty specific to the bus stop analysis. It's so specific, in fact, that while there are a few ways in which we can tweak it to make it more general (that is, useful in other scripts that might not have the same steps involved), we should not convert it into a separate function. When we create a separate function, we introduce too many variables into the script in an effort to simplify it, which is a counterproductive effort. Instead, let's focus on ways to generalize the ArcPy tools themselves. The first step will be to split the three ArcPy tools and examine what can be adjusted with each of them. The Select tool should be adjusted to accept a string as the SQL select statement. The SQL statement can then be generated by another function or by parameters accepted at runtime. For instance, if we wanted to make the script accept multiple bus stops for each run of the script (for example, the inbound and outbound stops for each line), we could create a function that would accept a list of the desired stops and a SQL template, and would return a SQL statement to plug into the Select tool. Here is an example of how it would look: def formatSQLIN(dataList, sqlTemplate):    'a function to generate a SQL statement'    sql = sqlTemplate #"OBJECTID IN "    step = "("    for data in dataList:        step += str(data)    sql += step + ")"    return sql   def formatSQL(dataList, sqlTemplate):    'a function to generate a SQL statement'    sql = ''    for count, data in enumerate(dataList):        if count != len(dataList)-1:            sql += sqlTemplate.format(data) + ' OR '        else:            sql += sqlTemplate.format(data)    return sql   >>> dataVals = [1,2,3,4] >>> sqlOID = "OBJECTID = {0}" >>> sql = formatSQL(dataVals, sqlOID) >>> print sql The output is as follows: OBJECTID = 1 OR OBJECTID = 2 OR OBJECTID = 3 OR OBJECTID = 4 This new function, formatSQL(), is a very useful function. Let's review what it does by comparing the function to the results following it. The function is defined to accept two parameters: a list of values and a SQL template. The first local variable is the empty string sql, which will be added to using string addition. The function is designed to insert the values into the variable sql, creating a SQL statement by taking the SQL template and using string formatting to add them to the template, which in turn is added to the SQL statement string (note that sql += is equivelent to sql = sql +). Also, an operator (OR) is used to make the SQL statement inclusive of all data rows that match the pattern. This function uses the built-in enumerate function to count the iterations of the list; once it has reached the last value in the list, the operator is not added to the SQL statement. Note that we could also add one more parameter to the function to make it possible to use an AND operator instead of OR, while still keeping OR as the default: def formatSQL2(dataList, sqlTemplate, operator=" OR "):    'a function to generate a SQL statement'    sql = ''    for count, data in enumerate(dataList):        if count != len(dataList)-1:            sql += sqlTemplate.format(data) + operator        else:            sql += sqlTemplate.format(data)    return sql   >>> sql = formatSQL2(dataVals, sqlOID," AND ") >>> print sql The output is as follows: OBJECTID = 1 AND OBJECTID = 2 AND OBJECTID = 3 AND OBJECTID = 4 While it would make no sense to use an AND operator on ObjectIDs, there are other cases where it would make sense, hence leaving OR as the default while allowing for AND. Either way, this function can now be used to generate our bus stop SQL statement for multiple stops (ignoring, for now, the bus signage field): >>> sqlTemplate = "NAME = '{0}'" >>> lineNames = ['71 IB','71 OB'] >>> sql = formatSQL2(lineNames, sqlTemplate) >>> print sql The output is as follows: NAME = '71 IB' OR NAME = '71 OB' However, we can't ignore the Bus Signage field for the inbound line, as there are two starting points for the line, so we will need to adjust the function to accept multiple values: def formatSQLMultiple(dataList, sqlTemplate, operator=" OR "):    'a function to generate a SQL statement'    sql = ''    for count, data in enumerate(dataList):        if count != len(dataList)-1:            sql += sqlTemplate.format(*data) + operator        else:            sql += sqlTemplate.format(*data)    return sql   >>> sqlTemplate = "(NAME = '{0}' AND BUS_SIGNAG = '{1}')" >>> lineNames = [('71 IB', 'Ferry Plaza'),('71 OB','48th Avenue')] >>> sql = formatSQLMultiple(lineNames, sqlTemplate) >>> print sql The output is as follows: (NAME = '71 IB' AND BUS_SIGNAG = 'Ferry Plaza') OR (NAME = '71 OB' AND BUS_SIGNAG = '48th Avenue') The slight difference in this function, the asterisk before the data variable, allows the values inside the data variable to be correctly formatted into the SQL template by exploding the values within the tuple. Notice that the SQL template has been created to segregate each conditional by using parentheses. The function(s) are now ready for reuse, and the SQL statement is now ready for insertion into the Select tool: sql = formatSQLMultiple(lineNames, sqlTemplate) arcpy.Select_analysis(Bus_Stops, Inbound71, sql) Next up is the Buffer tool. We have already taken steps towards making it generalized by adding a variable for the distance. In this case, we will only add one more variable to it, a unit variable that will make it possible to adjust the buffer unit from feet to meter or any other allowed unit. We will leave the other defaults alone. Here is an adjusted version of the Buffer tool: bufferDist = 400 bufferUnit = "Feet" arcpy.Buffer_analysis(Inbound71,                      Inbound71_400ft_buffer,                      "{0} {1}".format(bufferDist, bufferUnit),                      "FULL", "ROUND", "NONE", "") Now, both the buffer distance and buffer unit are controlled by a variable defined in the previous script, and this will make it easily adjustable if it is decided that the distance was not sufficient and the variables might need to be adjusted. The next step towards adjusting the ArcPy tools is to write a function, which will allow for any number of feature classes to be intersected together using the Intersect tool. This new function will be similar to the formatSQL functions as previous, as they will use string formatting and addition to allow for a list of feature classes to be processed into the correct string format for the Intersect tool to accept them. However, as this function will be built to be as general as possible, it must be designed to accept any number of feature classes to be intersected: def formatIntersect(features):    'a function to generate an intersect string'    formatString = ''    for count, feature in enumerate(features):        if count != len(features)-1:            formatString += feature + " #;"        else:            formatString += feature + " #"        return formatString >>> shpNames = ["example.shp","example2.shp"] >>> iString = formatIntersect(shpNames) >>> print iString The output is as follows: example.shp #;example2.shp # Now that we have written the formatIntersect() function, all that needs to be created is a list of the feature classes to be passed to the function. The string returned by the function can then be passed to the Intersect tool: intersected = [Inbound71_400ft_buffer, CensusBlocks2010] iString = formatIntersect(intersected) # Process: Intersect arcpy.Intersect_analysis(iString,                          Intersect71Census, "ALL", "", "INPUT") Because we avoided creating a function that only fits this script or analysis, we now have two (or more) useful functions that can be applied in later analyses, and we know how to manipulate the ArcPy tools to accept the data that we want to supply to them. Summary In this article, we discussed how to take autogenerated code and make it generalized, while adding functions that can be reused in other scripts and will make the generation of the necessary code components, such as SQL statements, much easier. Resources for Article: Further resources on this subject: Enterprise Geodatabase [article] Adding Graphics to the Map [article] Image classification and feature extraction from images [article]

It’s been over five decades since programming pushed the boundaries of digital craftsmanship, and it is still doing so with no signs of stopping or slowing down. There is a new tool, framework, add-on, functionality, technology, or a programming language breaking the Internet every now and then. Any adept programmer not only needs to be good at coding but also has to stay abreast with the ongoing and upcoming happenings in the programming world. Just learning to code does not a give you a big edge over others. By having a good idea of what’s coming ahead,  present steps can be planned effectively. Obviously, no one can perfectly forecast the future of computer programming, but that won’t stop us from speculating, right! Here are 11  predictions for the future of programming that we think programmers should keep an eye on. #1 Cloud native as the new default Do you know that in order to cater to a single search query, Google Search uses more than 1000s of servers? All this is done in order to serve the right results. Cloud has been popular for past one decade but it’s destined to grow immensely in the future as more and more developers intend to use cloud for faster go to market. Tinkering in the cloud to build an app is so much easier as compared to managing your own servers as you don’t have to buy new servers, maintain them, upgrade them, or add new servers as and when the demand fluctuates. Web users are an impatient lot these days; so making web pages faster is the main goal for developers. 40% of people abandon a website that takes more than 3 seconds to load. More efficient algorithms save a few microseconds whereas additional impetus is provided by the rapidly developing enhanced servers. #2 IoT security concerns will escalate IoT is a growing technological concept these days. The promising piece of tech has already made it to the market, although in a limited form. Any smart device is just like a computer or machine that can be hacked by means of feeding some simple malicious lines of code. So, security of IoT devices is as important as their deployment. Or else, we will have to face dire consequences, as experienced recently in the form of a North Korean hacker charged for WannaCry ransomware and a 16 year old hacking into Apple’s servers to access customer data. Programmers need to develop suspicious-activity-proof algorithms for IoT devices. Failing to do so will not only make the devices vulnerable to unintended use but also put the entire system at risk. Hence, with the growth in the IoT market, concern about its safety will also mushroom. #3 Video Content will continue to dominate the Web In order to solve the dire glitches caused by plugins, the HTML standards committee started embedding video tags into HTML. Videos tags are programmable by virtue of the fact that basic video tags respond to JavaScript commands. Earlier video content was fixed. If you watch a video about dogs fighting cats, then you will be recommended just that. Nothing more, nothing less. However, this is not the case anymore. It is the time of seamless canvas design, in which web designers figure out clever ways to deploy different video content. Doing so allows the user to steer the way in which a narrative is unfolded and it opens up new ways of interacting with the video content. Now machine learning can deliver higher-quality streaming experiences that do not buffer as much as many existing systems. More efficient codecs and better video compression are also playing a role in making video a better digital consumption medium. Again, programming makes it feasible, as video tags and iframe are part of the programming code. #4 Consoles, consoles everywhere Thanks to the groundbreaking progress in video game console technology, PCs are continuously being rejected in favor of gaming consoles. Living room consoles are just the start. With the concept of intelligent devices, makers of other household items are also looking to make their offerings smarter. Our hairdryers and toasters are already boasting digital memory, allowing for remembering our preferences. However, the time when these, and other household units as well, will start communicating with each other i.e. exchanging information on their own is yet to come. All of these scenarios are only made possible by programming. As several programmers have already embarked on the journey for achieving results in the same direction, we might not be that far away from a time when the aforementioned scenario would be a day-to-day reality. #5 Data is important, data will be important Data is the backbone of the network of networks i.e. the Internet. What we see, read, and hear over the gigantic web is data, loads and loads of it. However, data collection is not something new for humanity. Since antiquity, humans have collected and stored large chunks of data for churning out important information at some later time. With the passage of time, enriching and protecting data have become important. While the former is achieved by presenting data in the form of videos, pictures, pie charts, etc., the latter is accomplished by adding SSL to the website and using better encryption techniques. Data processing has become equally important just like the digital ecosphere itself. In the enterprise community, data gathering will branch out more elaborately into storing, curating, and parsing. Simply said, data is and data will be the undisputed champion in the Digital World. #6 Machine Learning dominance Machine Learning is already flourishing and seeping into everyday enterprise and life. For example, machine learning algorithms are already finding a place in important automation code for big businesses. They are used for heaping big data projects. Languages like the R programming language and Python have enabled this proliferation of machine learning, so far. What’s amazing about machine learning is that it is slowly being integrated into modern life. It will soon become a common entity in a person’s life, just like smartphones and IoT. Again, machine learning also requires services of programming and code, of course. No code, no machine learning. At least for now. There is the rise of machine learning as a service trend which aims to remove or minimize programming. However, if we ever learning anything from the history of web development, even as drag and drop web design tools grow, professional web developers also grow in demand. We can expect to see a similar trend with machine learning as it continues down the path of democratization. #7 User Interface design will continue gaining popularity The time when an Internet user was expected to use a keyboard and mouse is long gone. With each passing day, using a PC is preferred less and less. Apart from offices and college laboratories, PCs are gradually being replaced by other smart devices. As smartphones, tablets, living room consoles, etc. take on the world, the emphasis on UI has heightened. A touch and a click on the screen is different. With the advancement in technology, the former is given preference. This is because it’s quick and convenient at the same time. Furthermore, face and fingerprint recognition are the new cool. Research on voice control is also advancing. Many brands have already incepted their very own virtual assistants, such as Amazon Alexa, Siri and Google Assistant, which can recognize the demands of their users with mere voice commands and interaction. For example, Android 9 Pie comes with a number of UI alterations to stay relevant with the present UI scenario, including a new position for the volume controls and Material Theming. The latter is a built-in Android toolset meant for customizing the Material Design supported by the Android. Again, designing a powerful user interface is dependent on great programming. A user interface needs not only to be robust only but also show signs of intuitiveness and interactivity. The stress on UI designing will continue growing in the future. Some of the upcoming UI trends forecasted for 2019 are the overlapping effect, functional animations, and contrast of fonts. #8 Open Source vs. Closed Development Nearly all laptops run on proprietary software but Smartphones with Android leading the race are mostly open source. iOS is still closed but it has a robust set of APIs on which developers can build their own empires. While open source software is something that anyone can tinker with, closed development environment restricts 3rd-party from accessing and toying with such a system. Among other differences between the two, a significant difference is in the quality of support. This is, obviously, better offered by closed source software. Open source is rocking the world with new developers entering into programming by tinkering with open source whereas closed environment is also growing tremendously because of personalization and security features. This is one hell of a competition. #9 Autonomous Transportation Another industry that requires services of programming is the autonomous vehicles. Just yesterday, Waymo announced that their first driverless cars will be on the road commercially next month. So far, we have only seen some of the many accomplishments that a driverless mode of transportation can achieve. Though we have only cars, for now, that is making use of autonomous transportation algorithms, soon other transportation means will also join the parade. There are already crowdfunding projects for autonomous skateboards. Known as XTND Board, it is a lightweight electric vehicle meant to redefine commuting. Autonomous aircrafts are  already being used in the military. However, pilotless airplane transportation may just be around the corner. All it requires is an excellent programming code to allow a vehicle to know that what route should it chose. So, maybe flights might become autonomous after rides. #10 The Law will redefine new limits Writing code is like fixing something, setting up protocols. What the program will do and what it won’t, depends entirely on the coding. However, there are several ways to manipulate harmful programming code. There’s a subtle analogy between programming code and law and both have their own jurisdictions. Though there is a bright, sunny side to the technological advancement, there’s also a darker side of the same that needs to be reviewed and regulated. As years will pass from this point in time, programmers will face real-world challenges to assist the Law & Order to sustain the malicious content of the society, both on the digital front and the real-world front. We have already seen how adding technology to law works. However, the other side is that it can also act as a tool to break the law(s). Cyberattacks, identity theft, and data laundering are some of the notable examples made possible by technology. This is a question which is also its own solution. In order to prevent such insincere acts, security personnel need to think like bypassers. This is where ethical hacking comes in. It is simply thinking and operating like a malicious hacker but doing so for the right cause. #11 Containers will continue to rule Theoretically, there isn’t a need for the so-called containers, which are heavily deployed in the modern-day programming. In theory, the executable files can run anywhere and various requisite permissions, such as using hardware, are given by the OS. Hence, there is, theoretically, no requirements for a container. However, because of being theoretical, all executables are considered the same. Obviously, this is not the general case. What happens is that executables are different and each one of them requires specific libraries to run. For instance, the WORA (Write Once, Run Anywhere) chant of Java fails owing to the virtue that there are several different versions of virtual machines (VMs). Though using a comprehensive VM might solve the issue, the solution lacks practicality. On the other hand, the sleek and lightweight containers win the preference. Containers are the solution to the issue of reliability caused by a software when it is to be migrated from one computing environment to another. A container is simply a complete package that contains an entire runtime environment with the application, its dependencies and libraries, other required binaries, configuration files, etc. So, when a container of a specific application has everything in it that it requires to operate, the container becomes independent of the platform. The containers will continue to rule in the future up ahead. If you are new to programming, you can check out programming terms for beginners to kickstart your coding journey. These were the future predictions that we can think of. Do want to add anything else? Please feel free to do so in the comments below. Author Bio Saurabh has worked globally for telecom and finance giants in various capacities. After working for a decade in Infosys and Sapient, he started his first startup, Leno, to solve a hyperlocal book-sharing problem. He is interested in product marketing, and analytics. His latest venture Hackr.io recommends the best online programming and design courses for every programming language. All the tutorials are submitted and voted by the programming community. What we learned from IBM Research’s ‘5 in 5’ predictions presented at Think 2018. “Deep learning is not an optimum solution for every problem faced”: An interview with Valentino Zocca. Why does the C programming language refuse to die?  

The 2018 survey of the RedMonk Programming Language Rankings marked the entry of a new programming language in their Top 25 list. It has been an incredibly successful year for the Rust programming language in terms of its popularity. It also jumped from the 46th most popular language on GitHub to the 18th position. The Stack overflow survey of 2018 is another indicator of the rise of Rust programming language. Almost 78% of the developers who are working with Rust loved working on it. It topped the list of the most loved programming language among the developers who took the survey for a straight third year in the row. Not only that but it ranked 8th in the most wanted programming language in the survey, which means that the respondent of the survey who has not used it yet but would like to learn. Although, Rust was designed as a low-level language, best suited for systems, embedded, and other performance critical code, it is gaining a lot of traction and presents a great opportunity for web developers and game developers. RUST is also empowering novice developers with the tools to start shipping code fast. So, why is Rust so tempting? Let's explore the high points of this incredible language and understand the variety of features that make it interesting to learn. Automatic Garbage Collection Garbage collection and non-memory resources often create problems with some systems languages. But Rust pays no head to garbage collection and removes the possibilities of failures caused by them. In Rust, garbage collection is completely taken care of by RAII (Resource Acquisition Is Initialization). Better support for Concurrency Concurrency and parallelism are incredibly imperative topics in computer science and are also a hot topic in the industry today. Computers are gaining more and more cores, yet many programmers aren't prepared to fully utilize the power of them. Handling concurrent programming safely and efficiently is another major goal of Rust language. Concurrency is difficult to reason about. In Rust, there is a strong, static type system that helps to reason about your code. As such, Rust also gives you two traits Send and Sync to help you make sense of code that can possibly be concurrent. Rust's standard library also provides a library for threads, which enable you to run Rust code in parallel. You can also use Rust’s threads as a simple isolation mechanism. Error Handling in Rust is beautiful A programmer is bound to make errors, irrespective of the programming language they use. Making errors while programming is normal, but it's the error handling mechanism of that programming language, which enhances the experience of writing the code. In Rust, errors are divided into types: unrecoverable errors and recoverable errors. Unrecoverable errors An error is classified as 'unrecoverable' when there is no other option other than to abort the program. The panic! macro in Rust is very helpful in these cases, especially when a bug has been detected in the code but the programmer is not clear how to handle that error. The panic! macro generates a failure message that helps the user to debug a problem. It also helps to stop the execution before more catastrophic events occur. Recoverable errors The errors which can be handled easily or which do not have a serious impact on the execution of the program are known as recoverable errors. It is represented by the Result<T, E>. The Result<T, E> is an enum that consists of two variants, i.e., OK<T> and Err<E>. It describes the possible error in the program. OK<T>: The 'T' is a type of value which returns the OK variant in the success case. It is an expected outcome. Err<E>: The 'E' is a type of error which returns the ERR variant in the failure. It is an unexpected outcome. Resource Management The one attribute that makes Rust stand out (and completely overpowers Google’s Go for that matter), is the algorithm used for resource management. Rust follows the C++ lead, with concepts like borrowing and mutable borrowing on the plate and thus resource management becomes an elegant process. Furthermore, Rust didn’t need a second chance to know that resource management is not just about memory usage; the fact that they did it right first time makes them a standout performer on this point. Although the Rust documentation does a good job of explaining the technical details, the article by Tim explains the concept in a much friendlier and easy to understand language. As such I thought, it would be good to list his points as well here. The following excerpt is taken from the article written by M.Tim Jones. Reusable code via modules Rust allows you to organize code in a way that promotes its reuse. You attain this reusability by using modules which are nothing but organized code as packages that other programmers can use. These modules contain functions, structures and even other modules that you can either make public, which can be accessed by the users of the module or you can make it private which can be used only within the module and not by the module user. There are three keywords to create modules, use modules, and modify the visibility of elements in modules. The mod keyword creates a new module The use keyword allows you to use the module (expose the definitions into the scope to use them) The pub keyword makes elements of the module public (otherwise, they're private). Cleaner code with better safety checks In Rust, the compiler enforces memory safety and another checking that make the programming language safe. Here, you will never have to worry about dangling pointers or bother using an object after it has been freed. These things are part of the core Rust language that allows you to write clean code. Also, Rust includes an unsafe keyword with which you can disable checks that would typically result in a compilation error. Data types and Collections in Rust Rust is a statically typed programming language, which means that every value in Rust must have a specified data type. The biggest advantage of static typing is that a large class of errors is identified earlier in the development process. These data types can be broadly classified into two types: scalar and compound. Scalar data types represent a single value like integer, floating-point, and character, which are commonly present in other programming languages as well. But Rust also provides compound data types which allow the programmers to group multiple values in one type such as tuples and arrays. The Rust standard library provides a number of data structures which are also called collections. Collections contain multiple values but they are different from the standard compound data types like tuples and arrays which we discussed above. The biggest advantage of using collections is the capability of not specifying the amount of data at compile time which allows the structure to grow and shrink as the program runs. Vectors, Strings, and hash maps are the three most commonly used collections in Rust. The friendly Rust community Rust owes it success to the breadth and depth of engagement of its vibrant community, which supports a highly collaborative process for helping the language to evolve in a truly open-source way. Rust is built from the bottom up, rather than any one individual or organization controlling the fate of the technology. Reliable Robust Release cycles of Rust What is common between Java, Spring, and Angular? They never release their update when they promise to. The release cycle of the Rust community works with clockwork precision and is very reliable. Here’s an overview of the dates and versions: In mid-September 2018, the Rust team released Rust 2018 RC1 version. Rust 2018 is the first major new edition of Rust (after Rust 1.0 released in 2015). This new release would mark the culmination of the last three years of Rust’s development from the core team, and brings the language together in one neat package. This version includes plenty of new features like raw identifiers, better path clarity, new optimizations, and other additions. You can learn more about the Rust language and its evolution at the Rust blog and download from the Rust language website. Note: the headline was edited 09.06.2018 to make it clear that Rust was found to be the most loved language among developers using it. Rust 2018 RC1 now released with Raw identifiers, better path clarity, and other changes Rust as a Game Programming Language: Is it any good? Rust Language Server, RLS 1.0 releases with code intelligence, syntax highlighting and more

The Scala Plugin is used to turn a normal IntelliJ IDEA into a convenient Scala development environment. In this article, we will discuss how to set up Scala Plugin for IntelliJ IDEA IDE.  If you do not have IntelliJ IDEA, you can download it from here. By default, IntelliJ IDEA does not come with Scala features. Scala Plugin adds Scala features means that we can create Scala/Play Projects, we can create Scala Applications, Scala worksheets, and more. Scala Plugin contains the following technologies: Scala Play Framework SBT Scala.js It supports three popular OS Environments: Windows, Mac, and Linux. Setting up Scala Plugin for IntelliJ IDE Perform the  following steps to install Scala Plugin for IntelliJ IDE to develop our Scala-based projects: Open IntelliJ IDE: Go to  Configure at the bottom right and click on the Plugins option available in the drop-down, as shown here: This opens the Plugins window as shown here: Now click on InstallJetbrainsplugins, as shown in the preceding screenshot. Next, type the word Scala in the search bar to see the ScalaPlugin, as shown here: Click on the Install button to install Scala Plugin for IntelliJ IDEA. Now restart IntelliJ IDEA to see that Scala Plugin features. After we re-open IntelliJ IDEA, if we try to access File | New Project option, we will see Scala option in New Project window as shown in the following screenshot to create new Scala or Play Framework-based SBT projects: We can see the Play Framework option only in the IntelliJ IDEA Ultimate Edition. As we are using CE (Community Edition), we cannot see that option. It's now time to start Scala/Play application development using the IntelliJ IDE. You can start developing some Scala/Play-based applications. To summarize, we got an understanding to Scala Plugin and covered the installation steps for Scala Plugin for IntelliJ. To learn more about solutions for taking reactive programming approach with Scala, please refer the book Scala Reactive Programming. What Scala 3.0 Roadmap looks like! Building Scalable Microservices Exploring Scala Performance

In this tutorial, we'll see how common design patterns can be used as blueprints for organizing larger structures. Defining steps with template functions A template is a design pattern that details the order a given set of operations are to be executed in; however, a template does not outline the steps themselves. This pattern is useful when behavior is divided into phases that have some conceptual or side effect dependency that requires them to be executed in a specific order. Here, we'll see how to use the template function design pattern. We assume you already have a workspace that allows you to create and run ES modules in your browser for all the recipes given below: How to do it... Open your command-line application and navigate to your workspace. Create a new folder named 09-01-defining-steps-with-template-functions. Copy or create an index.html file that loads and runs a main function from main.js. Create a main.js file that defines a new abstract class named Mission: // main.js class Mission { constructor () { if (this.constructor === Mission) { throw new Error('Mission is an abstract class, must extend'); } } } Add a function named execute that calls three instance methods—determineDestination, determinPayload, and launch: // main.js class Mission { execute () { this.determinDestination(); this.determinePayload(); this.launch(); } } Create a LunarRover class that extends the Mission class: // main.js class LunarRover extends Mission {} Add a constructor that assigns name to an instance property: // main.js class LunarRover extends Mission constructor (name) { super(); this.name = name; } } Implement the three methods called by Mission.execute: // main.js class LunarRover extends Mission {} determinDestination() { this.destination = 'Oceanus Procellarum'; } determinePayload() { this.payload = 'Rover with camera and mass spectrometer.'; } launch() { console.log(` Destination: ${this.destination} Playload: ${this.payload} Lauched! Rover Will arrive in a week. `); } } Create a JovianOrbiter class that also extends the Mission class: // main.js class LunarRover extends Mission {} constructor (name) { super(); this.name = name; } determinDestination() { this.destination = 'Jovian Orbit'; } determinePayload() { this.payload = 'Orbiter with decent module.'; } launch() { console.log(` Destination: ${this.destination} Playload: ${this.payload} Lauched! Orbiter Will arrive in 7 years. `); } } Create a main function that creates both concrete mission types and executes them: // main.js export function main() { const jadeRabbit = new LunarRover('Jade Rabbit'); jadeRabbit.execute(); const galileo = new JovianOrbiter('Galileo'); galileo.execute(); } Start your Python web server and open the following link in your browser: http://localhost:8000/. The output should appear as follows: How it works... The Mission abstract class defines the execute method, which calls the other instance methods in a particular order. You'll notice that the methods called are not defined by the Mission class. This implementation detail is the responsibility of the extending classes. This use of abstract classes allows child classes to be used by code that takes advantage of the interface defined by the abstract class. In the template function pattern, it is the responsibility of the child classes to define the steps. When they are instantiated, and the execute method is called, those steps are then performed in the specified order. Ideally, we'd be able to ensure that Mission.execute was not overridden by any inheriting classes. Overriding this method works against the pattern and breaks the contract associated with it. This pattern is useful for organizing data-processing pipelines. The guarantee that these steps will occur in a given order means that, if side effects are eliminated, the instances can be organized more flexibly. The implementing class can then organize these steps in the best possible way. Assembling customized instances with builders The previous recipe shows how to organize the operations of a class. Sometimes, object initialization can also be complicated. In these situations, it can be useful to take advantage of another design pattern: builders. Now, we'll see how to use builders to organize the initialization of more complicated objects. How to do it... Open your command-line application and navigate to your workspace. Create a new folder named 09-02-assembling-instances-with-builders. Create a main.js file that defines a new class named Mission, which that takes a name constructor argument and assigns it to an instance property. Also, create a describe method that prints out some details: // main.js class Mission { constructor (name) { this.name = name; } describe () { console.log(` The ${this.name} mission will be launched by a ${this.rocket.name} rocket, and deliver a ${this.payload.name} to ${this.destination.name}. `); } } Create classes named Destination, Payload, and Rocket, which receive a name property as a constructor parameter and assign it to an instance property: // main.js class Destination { constructor (name) { this.name = name; } } class Payload { constructor (name) { this.name = name; } } class Rocket { constructor (name) { this.name = name; } }   Create a MissionBuilder class that defines the setMissionName, setDestination, setPayload, and setRocket methods: // main.js class MissionBuilder { setMissionName (name) { this.missionName = name; return this; } setDestination (destination) { this.destination = destination; return this; } setPayload (payload) { this.payload = payload; return this; } setRocket (rocket) { this.rocket = rocket; return this; } } Create a build method that creates a new Mission instance with the appropriate properties: // main.js class MissionBuilder { build () { const mission = new Mission(this.missionName); mission.rocket = this.rocket; mission.destination = this.destination; mission.payload = this.payload; return mission; } } Create a main function that uses MissionBuilder to create a new mission instance: // main.js export function main() { // build an describe a mission new MissionBuilder() .setMissionName('Jade Rabbit') .setDestination(new Destination('Oceanus Procellarum')) .setPayload(new Payload('Lunar Rover')) .setRocket(new Rocket('Long March 3B Y-23')) .build() .describe(); } Start your Python web server and open the following link in your browser: http://localhost:8000/. Your output should appear as follows: How it works... The builder defines methods for assigning all the relevant properties and defines a build method that ensures that each is called and assigned appropriately. Builders are like template functions, but instead of ensuring that a set of operations are executed in the correct order, they ensure that an instance is properly configured before returning. Because each instance method of MissionBuilder returns the this reference, the methods can be chained. The last line of the main function calls describe on the new Mission instance that is returned from the build method. Replicating instances with factories Like builders, factories are a way of organizing object construction. They differ from builders in how they are organized. Often, the interface of factories is a single function call. This makes factories easier to use, if less customizable, than builders. Now, we'll see how to use factories to easily replicate instances. How to do it... Open your command-line application and navigate to your workspace. Create a new folder named 09-03-replicating-instances-with-factories. Copy or create an index.html that loads and runs a main function from main.js. Create a main.js file that defines a new class named Mission. Add a constructor that takes a name constructor argument and assigns it to an instance property. Also, define a simple describe method: // main.js class Mission { constructor (name) { this.name = name; } describe () { console.log(` The ${this.name} mission will be launched by a ${this.rocket.name} rocket, and deliver a ${this.payload.name} to ${this.destination.name}. `); } } Create three classes named Destination, Payload, and Rocket, that take name as a constructor argument and assign it to an instance property: // main.js class Destination { constructor (name) { this.name = name; } } class Payload { constructor (name) { this.name = name; } } class Rocket { constructor (name) { this.name = name; } } Create a MarsMissionFactory object with a single create method that takes two arguments: name and rocket. This method should create a new Mission using those arguments: // main.js const MarsMissionFactory = { create (name, rocket) { const mission = new Mission(name); mission.destination = new Destination('Martian surface'); mission.payload = new Payload('Mars rover'); mission.rocket = rocket; return mission; } } Create a main method that creates and describes two similar missions: // main.js export function main() { // build an describe a mission MarsMissionFactory .create('Curiosity', new Rocket('Atlas V')) .describe(); MarsMissionFactory .create('Spirit', new Rocket('Delta II')) .describe(); } Start your Python web server and open the following link in your browser: http://localhost:8000/. Your output should appear as follows: How it works... The create method takes a subset of the properties needed to create a new mission. The remaining values are provided by the method itself. This allows factories to simplify the process of creating similar instances. In the main function, you can see that two Mars missions have been created, only differing in name and Rocket instance. We've halved the number of values needed to create an instance. This pattern can help reduce instantiation logic. In this recipe, we simplified the creation of different kinds of missions by identifying the common attributes, encapsulating those in the body of the factory function, and using arguments to supply the remaining properties. In this way, commonly used instance shapes can be created without additional boilerplate code. Processing a structure with the visitor pattern The patterns we've seen thus far organize the construction of objects and the execution of operations. The next pattern we'll look at is specially made to traverse and perform operations on hierarchical structures. Here, we'll be looking at the visitor pattern. How to do it... Open your command-line application and navigate to your workspace. Copy the 09-02-assembling-instances-with-builders folder to a new 09-04-processing-a-structure-with-the-visitor-pattern directory. Add a class named MissionInspector to main.js. Create a visitor method that calls a corresponding method for each of the following types: Mission, Destination, Rocket, and Payload: // main.js /* visitor that inspects mission */ class MissionInspector { visit (element) { if (element instanceof Mission) { this.visitMission(element); } else if (element instanceof Destination) { this.visitDestination(element); } else if (element instanceof Rocket) { this.visitRocket(element); } else if (element instanceof Payload) { this.visitPayload(element); } } } Create a visitMission method that logs out an ok message: // main.js class MissionInspector { visitMission (mission) { console.log('Mission ok'); mission.describe(); } } Create a visitDestination method that throws an error if the destination is not in an approved list: // main.js class MissionInspector { visitDestination (destination) { const name = destination.name.toLowerCase(); if ( name === 'mercury' || name === 'venus' || name === 'earth' || name === 'moon' || name === 'mars' ) { console.log('Destination: ', name, ' approved'); } else { throw new Error('Destination: '' + name + '' not approved at this time'); } } } Create a visitPayload method that throws an error if the payload isn't valid: // main.js class MissionInspector { visitPayload (payload) { const name = payload.name.toLowerCase(); const payloadExpr = /(orbiter)|(rover)/; if ( payloadExpr.test(name) ) { console.log('Payload: ', name, ' approved'); } else { throw new Error('Payload: '' + name + '' not approved at this time'); } } } Create a visitRocket method that logs out an ok message: // main.js class MissionInspector { visitRocket (rocket) { console.log('Rocket: ', rocket.name, ' approved'); } } Add an accept method to the Mission class that calls accept on its constituents, then tells visitor to visit the current instance: // main.js class Mission { // other mission code ... accept (visitor) { this.rocket.accept(visitor); this.payload.accept(visitor); this.destination.accept(visitor); visitor.visit(this); } } Add an accept method to the Destination class that tells visitor to visit the current instance: // main.js class Destination { // other mission code ... accept (visitor) { visitor.visit(this); } } Add an accept method to the Payload class that tells visitor to visit the current instance: // main.js class Payload { // other mission code ... accept (visitor) { visitor.visit(this); } } Add an accept method to the Rocket class that tells visitor to visit the current instance: // main.js class Rocket { // other mission code ... accept (visitor) { visitor.visit(this); } } Create a main function that creates different instances with the builder, visits them with the MissionInspector instance, and logs out any thrown errors: // main.js export function main() { // build an describe a mission const jadeRabbit = new MissionBuilder() .setMissionName('Jade Rabbit') .setDestination(new Destination('Moon')) .setPayload(new Payload('Lunar Rover')) .setRocket(new Rocket('Long March 3B Y-23')) .build(); const curiosity = new MissionBuilder() .setMissionName('Curiosity') .setDestination(new Destination('Mars')) .setPayload(new Payload('Mars Rover')) .setRocket(new Rocket('Delta II')) .build(); // expect error from Destination const buzz = new MissionBuilder() .setMissionName('Buzz Lightyear') .setDestination(new Destination('Too Infinity And Beyond')) .setPayload(new Payload('Interstellar Orbiter')) .setRocket(new Rocket('Self Propelled')) .build(); // expect error from payload const terraformer = new MissionBuilder() .setMissionName('Mars Terraformer') .setDestination(new Destination('Mars')) .setPayload(new Payload('Terraformer')) .setRocket(new Rocket('Light Sail')) .build(); const inspector = new MissionInspector(); [jadeRabbit, curiosity, buzz, terraformer].forEach((mission) => { try { mission.accept(inspector); } catch (e) { console.error(e); } }); } Start your Python web server and open the following link in your browser: http://localhost:8000/. Your output should appear as follows: How it works... The visitor pattern has two components. The visitor processes the subject objects and the subjects tell other related subjects about the visitor, and when the current subject should be visited. The accept method is required for each subject to receive a notification that there is a visitor. That method then makes two types of method call. The first is the accept method on its related subjects. The second is the visitor method on the visitor. In this way, the visitor traverses a structure by being passed around by the subjects. The visitor methods are used to process different types of node. In some languages, this is handled by language-level polymorphism. In JavaScript, we can use run-time type checks to do this. The visitor pattern is a good option for processing hierarchical structures of objects, where the structure is not known ahead of time, but the types of subjects are known. Using a singleton to manage instances Sometimes, there are objects that are resource intensive. They may require time, memory, battery power, or network usage that are unavailable or inconvenient. It is often useful to manage the creation and sharing of instances. Here, we'll see how to use singletons to manage instances. How to do it... Open your command-line application and navigate to your workspace. Create a new folder named 09-05-singleton-to-manage-instances. Copy or create an index.html that loads and runs a main function from main.js. Create a main.js file that defines a new class named Rocket. Add a constructor takes a name constructor argument and assigns it to an instance property: // main.js class Rocket { constructor (name) { this.name = name; } } Create a RocketManager object that has a rockets property. Add a findOrCreate method that indexes Rocket instances by the name property: // main.js const RocketManager = { rockets: {}, findOrCreate (name) { const rocket = this.rockets[name] || new Rocket(name); this.rockets[name] = rocket; return rocket; } } Create a main function that creates instances with and without the manager. Compare the instances and see whether they are identical: // main.js export function main() { const atlas = RocketManager.findOrCreate('Atlas V'); const atlasCopy = RocketManager.findOrCreate('Atlas V'); const atlasClone = new Rocket('Atlas V'); console.log('Copy is the same: ', atlas === atlasCopy); console.log('Clone is the same: ', atlas === atlasClone); } Start your Python web server and open the following link in your browser: http://localhost:8000/. Your output should appear as follows: How it works... The object stores references to the instances, indexed by the string value given with name. This map is created when the module loads, so it is persisted through the life of the program. The singleton is then able to look up the object and returns instances created by findOrCreate with the same name. Conserving resources and simplifying communication are primary motivations for using singletons. Creating a single object for multiple uses is more efficient in terms of space and time needed than creating several. Plus, having single instances for messages to be communicated through makes communication between different parts of a program easier. Singletons may require more sophisticated indexing if they are relying on more complicated data. You read an excerpt from a book written by Ross Harrison, titled ECMAScript Cookbook. This book contains over 70 recipes to help you improve your coding skills and solving practical JavaScript problems. 6 JavaScript micro optimizations you need to know Mozilla is building a bridge between Rust and JavaScript Behavior Scripting in C# and Javascript for game developers  

Creational design patterns deal with an object creation [j.mp/wikicrea]. The aim of a creational design pattern is to provide better alternatives for situations where a direct object creation (which in Python happens by the __init__() function [j.mp/divefunc], [Lott14, page 26]) is not convenient. In the Factory design pattern, a client asks for an object without knowing where the object is coming from (that is, which class is used to generate it). The idea behind a factory is to simplify an object creation. It is easier to track which objects are created if this is done through a central function, in contrast to letting a client create objects using a direct class instantiation [Eckel08, page 187]. A factory reduces the complexity of maintaining an application by decoupling the code that creates an object from the code that uses it [Zlobin13, page 30]. Factories typically come in two forms: the Factory Method, which is a method (or in Pythonic terms, a function) that returns a different object per input parameter [j.mp/factorympat]; the Abstract Factory, which is a group of Factory Methods used to create a family of related products [GOF95, page 100], [j.mp/absfpat] (For more resources related to this topic, see here.) Factory Method In the Factory Method, we execute a single function, passing a parameter that provides information about what we want. We are not required to know any details about how the object is implemented and where it is coming from. A real-life example An example of the Factory Method pattern used in reality is in plastic toy construction. The molding powder used to construct plastic toys is the same, but different figures can be produced using different plastic molds. This is like having a Factory Method in which the input is the name of the figure that we want (soldier and dinosaur) and the output is the plastic figure that we requested. The toy construction case is shown in the following figure, which is provided by www.sourcemaking.com [j.mp/factorympat]. A software example The Django framework uses the Factory Method pattern for creating the fields of a form. The forms module of Django supports the creation of different kinds of fields (CharField, EmailField) and customizations (max_length, required) [j.mp/djangofacm]. Use cases If you realize that you cannot track the objects created by your application because the code that creates them is in many different places instead of a single function/method, you should consider using the Factory Method pattern [Eckel08, page 187]. The Factory Method centralizes an object creation and tracking your objects becomes much more easier. Note that it is absolutely fine to create more than one Factory Method, and this is how it is typically done in practice. Each Factory Method logically groups the creation of objects that have similarities. For example, one Factory Method might be responsible for connecting you to different databases (MySQL, SQLite), another Factory Method might be responsible for creating the geometrical object that you request (circle, triangle), and so on. The Factory Method is also useful when you want to decouple an object creation from an object usage. We are not coupled/bound to a specific class when creating an object, we just provide partial information about what we want by calling a function. This means that introducing changes to the function is easy without requiring any changes to the code that uses it [Zlobin13, page 30]. Another use case worth mentioning is related with improving the performance and memory usage of an application. A Factory Method can improve the performance and memory usage by creating new objects only if it is absolutely necessary [Zlobin13, page 28]. When we create objects using a direct class instantiation, extra memory is allocated every time a new object is created (unless the class uses caching internally, which is usually not the case). We can see that in practice in the following code (file id.py), it creates two instances of the same class A and uses the id() function to compare their memory addresses. The addresses are also printed in the output so that we can inspect them. The fact that the memory addresses are different means that two distinct objects are created as follows: class A(object):     pass if __name__ == '__main__':     a = A()     b = A()     print(id(a) == id(b))     print(a, b) Executing id.py on my computer gives the following output:>> python3 id.pyFalse<__main__.A object at 0x7f5771de8f60> <__main__.A object at 0x7f5771df2208> Note that the addresses that you see if you execute the file are not the same as I see because they depend on the current memory layout and allocation. But the result must be the same: the two addresses should be different. There's one exception that happens if you write and execute the code in the Python Read-Eval-Print Loop (REPL) (interactive prompt), but that's a REPL-specific optimization which is not happening normally. Implementation Data comes in many forms. There are two main file categories for storing/retrieving data: human-readable files and binary files. Examples of human-readable files are XML, Atom, YAML, and JSON. Examples of binary files are the .sq3 file format used by SQLite and the .mp3 file format used to listen to music. In this example, we will focus on two popular human-readable formats: XML and JSON. Although human-readable files are generally slower to parse than binary files, they make data exchange, inspection, and modification much more easier. For this reason, it is advised to prefer working with human-readable files, unless there are other restrictions that do not allow it (mainly unacceptable performance and proprietary binary formats). In this problem, we have some input data stored in an XML and a JSON file, and we want to parse them and retrieve some information. At the same time, we want to centralize the client's connection to those (and all future) external services. We will use the Factory Method to solve this problem. The example focuses only on XML and JSON, but adding support for more services should be straightforward. First, let's take a look at the data files. The XML file, person.xml, is based on the Wikipedia example [j.mp/wikijson] and contains information about individuals (firstName, lastName, gender, and so on) as follows: <persons>   <person>     <firstName>John</firstName>     <lastName>Smith</lastName>     <age>25</age>     <address>       <streetAddress>21 2nd Street</streetAddress>       <city>New York</city>       <state>NY</state>       <postalCode>10021</postalCode>     </address>     <phoneNumbers>       <phoneNumber type="home">212 555-1234</phoneNumber>       <phoneNumber type="fax">646 555-4567</phoneNumber>     </phoneNumbers>     <gender>       <type>male</type>     </gender>   </person>   <person>     <firstName>Jimy</firstName>     <lastName>Liar</lastName>     <age>19</age>     <address>       <streetAddress>18 2nd Street</streetAddress>       <city>New York</city>       <state>NY</state>       <postalCode>10021</postalCode>     </address>     <phoneNumbers>       <phoneNumber type="home">212 555-1234</phoneNumber>     </phoneNumbers>     <gender>       <type>male</type>     </gender>   </person>   <person>     <firstName>Patty</firstName>     <lastName>Liar</lastName>     <age>20</age>     <address>       <streetAddress>18 2nd Street</streetAddress>       <city>New York</city>       <state>NY</state>       <postalCode>10021</postalCode>     </address>     <phoneNumbers>       <phoneNumber type="home">212 555-1234</phoneNumber>       <phoneNumber type="mobile">001 452-8819</phoneNumber>     </phoneNumbers>     <gender>       <type>female</type>     </gender>   </person> </persons> The JSON file, donut.json, comes from the GitHub account of Adobe [j.mp/adobejson] and contains donut information (type, price/unit i.e. ppu, topping, and so on) as follows: [   {     "id": "0001",     "type": "donut",     "name": "Cake",     "ppu": 0.55,     "batters": {       "batter": [         { "id": "1001", "type": "Regular" },         { "id": "1002", "type": "Chocolate" },         { "id": "1003", "type": "Blueberry" },         { "id": "1004", "type": "Devil's Food" }       ]     },     "topping": [       { "id": "5001", "type": "None" },       { "id": "5002", "type": "Glazed" },       { "id": "5005", "type": "Sugar" },       { "id": "5007", "type": "Powdered Sugar" },       { "id": "5006", "type": "Chocolate with Sprinkles" },       { "id": "5003", "type": "Chocolate" },       { "id": "5004", "type": "Maple" }     ]   },   {     "id": "0002",     "type": "donut",     "name": "Raised",     "ppu": 0.55,     "batters": {       "batter": [         { "id": "1001", "type": "Regular" }       ]     },     "topping": [       { "id": "5001", "type": "None" },       { "id": "5002", "type": "Glazed" },       { "id": "5005", "type": "Sugar" },       { "id": "5003", "type": "Chocolate" },       { "id": "5004", "type": "Maple" }     ]   },   {     "id": "0003",     "type": "donut",     "name": "Old Fashioned",     "ppu": 0.55,     "batters": {       "batter": [         { "id": "1001", "type": "Regular" },         { "id": "1002", "type": "Chocolate" }       ]     },     "topping": [       { "id": "5001", "type": "None" },       { "id": "5002", "type": "Glazed" },       { "id": "5003", "type": "Chocolate" },       { "id": "5004", "type": "Maple" }     ]   } ] We will use two libraries that are part of the Python distribution for working with XML and JSON: xml.etree.ElementTree and json as follows: import xml.etree.ElementTree as etree import json The JSONConnector class parses the JSON file and has a parsed_data() method that returns all data as a dictionary (dict). The property decorator is used to make parsed_data() appear as a normal variable instead of a method as follows: class JSONConnector:     def __init__(self, filepath):         self.data = dict()         with open(filepath, mode='r', encoding='utf-8') as f:             self.data = json.load(f)       @property     def parsed_data(self):         return self.data The XMLConnector class parses the XML file and has a parsed_data() method that returns all data as a list of xml.etree.Element as follows: class XMLConnector:     def __init__(self, filepath):         self.tree = etree.parse(filepath)     @property    def parsed_data(self):         return self.tree The connection_factory() function is a Factory Method. It returns an instance of JSONConnector or XMLConnector depending on the extension of the input file path as follows: def connection_factory(filepath):     if filepath.endswith('json'):         connector = JSONConnector     elif filepath.endswith('xml'):         connector = XMLConnector     else:         raise ValueError('Cannot connect to {}'.format(filepath))     return connector(filepath) The connect_to() function is a wrapper of connection_factory(). It adds exception handling as follows: def connect_to(filepath):     factory = None     try:         factory = connection_factory(filepath)     except ValueError as ve:         print(ve)     return factory The main() function demonstrates how the Factory Method design pattern can be used. The first part makes sure that exception handling is effective as follows: def main():     sqlite_factory = connect_to('data/person.sq3') The next part shows how to work with the XML files using the Factory Method. XPath is used to find all person elements that have the last name Liar. For each matched person, the basic name and phone number information are shown as follows: xml_factory = connect_to('data/person.xml')     xml_data = xml_factory.parsed_data()     liars = xml_data.findall     (".//{person}[{lastName}='{}']".format('Liar'))     print('found: {} persons'.format(len(liars)))     for liar in liars:         print('first name:         {}'.format(liar.find('firstName').text))         print('last name: {}'.format(liar.find('lastName').text))         [print('phone number ({}):'.format(p.attrib['type']),         p.text) for p in liar.find('phoneNumbers')] The final part shows how to work with the JSON files using the Factory Method. Here, there's no pattern matching, and therefore the name, price, and topping of all donuts are shown as follows: json_factory = connect_to('data/donut.json')     json_data = json_factory.parsed_data     print('found: {} donuts'.format(len(json_data)))     for donut in json_data:         print('name: {}'.format(donut['name']))         print('price: ${}'.format(donut['ppu']))         [print('topping: {} {}'.format(t['id'], t['type'])) for t         in donut['topping']] For completeness, here is the complete code of the Factory Method implementation (factory_method.py) as follows: import xml.etree.ElementTree as etree import json class JSONConnector:     def __init__(self, filepath):         self.data = dict()         with open(filepath, mode='r', encoding='utf-8') as f:             self.data = json.load(f)     @property     def parsed_data(self):         return self.data class XMLConnector:     def __init__(self, filepath):         self.tree = etree.parse(filepath)     @property     def parsed_data(self):         return self.tree def connection_factory(filepath):     if filepath.endswith('json'):         connector = JSONConnector     elif filepath.endswith('xml'):         connector = XMLConnector     else:         raise ValueError('Cannot connect to {}'.format(filepath))     return connector(filepath) def connect_to(filepath):     factory = None     try:        factory = connection_factory(filepath)     except ValueError as ve:         print(ve)     return factory def main():     sqlite_factory = connect_to('data/person.sq3')     print()     xml_factory = connect_to('data/person.xml')     xml_data = xml_factory.parsed_data     liars = xml_data.findall(".//{}[{}='{}']".format('person',     'lastName', 'Liar'))     print('found: {} persons'.format(len(liars)))     for liar in liars:         print('first name:         {}'.format(liar.find('firstName').text))         print('last name: {}'.format(liar.find('lastName').text))         [print('phone number ({}):'.format(p.attrib['type']),         p.text) for p in liar.find('phoneNumbers')]     print()     json_factory = connect_to('data/donut.json')     json_data = json_factory.parsed_data     print('found: {} donuts'.format(len(json_data)))     for donut in json_data:     print('name: {}'.format(donut['name']))     print('price: ${}'.format(donut['ppu']))     [print('topping: {} {}'.format(t['id'], t['type'])) for t     in donut['topping']] if __name__ == '__main__':     main() Here is the output of this program as follows: >>> python3 factory_method.pyCannot connect to data/person.sq3found: 2 personsfirst name: Jimylast name: Liarphone number (home): 212 555-1234first name: Pattylast name: Liarphone number (home): 212 555-1234phone number (mobile): 001 452-8819found: 3 donutsname: Cakeprice: $0.55topping: 5001 Nonetopping: 5002 Glazedtopping: 5005 Sugartopping: 5007 Powdered Sugartopping: 5006 Chocolate with Sprinklestopping: 5003 Chocolatetopping: 5004 Maplename: Raisedprice: $0.55topping: 5001 Nonetopping: 5002 Glazedtopping: 5005 Sugartopping: 5003 Chocolatetopping: 5004 Maplename: Old Fashionedprice: $0.55topping: 5001 Nonetopping: 5002 Glazedtopping: 5003 Chocolatetopping: 5004 Maple Notice that although JSONConnector and XMLConnector have the same interfaces, what is returned by parsed_data() is not handled in a uniform way. Different codes must be used to work with each connector. Although it would be nice to be able to use the same code for all connectors, this is at most times not realistic unless we use some kind of common mapping for the data which is very often provided by external data providers. Assuming that you can use exactly the same code for handling the XML and JSON files, what changes are required to support a third format, for example, SQLite? Find an SQLite file or create your own and try it. As is now, the code does not forbid a direct instantiation of a connector. Is it possible to do this? Try doing it (hint: functions in Python can have nested classes). Summary To learn more about design patterns in depth, the following books published by Packt Publishing (https://www.packtpub.com/) are recommended: Learning Python Design Patterns (https://www.packtpub.com/application-development/learning-python-design-patterns) Learning Python Design Patterns – Second Edition (https://www.packtpub.com/application-development/learning-python-design-patterns-second-edition) Resources for Article:   Further resources on this subject: Recommending Movies at Scale (Python) [article] An In-depth Look at Ansible Plugins [article] Elucidating the Game-changing Phenomenon of the Docker-inspired Containerization Paradigm [article]

(For more resources related to this topic, see here.) Neo4j is a graph database, which means that it does not use tables and rows to represent data logically; instead, it uses nodes and relationships. Both nodes and relationships can have a number of properties. While relationships must have one direction and one type, nodes can have a number of labels. For example, the following diagram shows three nodes and their relationships, where every node has a label (language or graph database), while relationships have a type (QUERY_LANGUAGE_OF and WRITTEN_IN). The properties used in the graph shown in the following diagram are: name, type, and from. Note that every relation must have exactly one type and one direction, whereas labels for nodes are optional and can be multiple. Neo4j running modes Neo4j can be used in two modes: An embedded database in a Java application; A standalone server via REST In any case, this choice does not affect the way you query and work with the database. It's only an architectural choice driven by the nature of the application (whether a standalone server or a client-server), performance, monitoring, and safety of data. An embedded database An embedded Neo4j database is the best choice for performance. It runs in the same process of the client application that hosts it and stores data in the given path. Thus, an embedded database must be created programmatically. We choose an embedded database for the following reasons: When we use Java as the programming language for our project When our application is standalone Preparing the development environment The fastest way to prepare the IDE for Neo4j is using Maven. Maven is a dependency management and automated building tool. In the following procedure, we will use NetBeans 7.4, but it works in a very similar way with the other IDEs (for Eclipse, you would need the m2eclipse plugin). The procedure is described as follows: Create a new Maven project as shown in the following screenshot: In the next page of the wizard, name the project, set a valid project location, and then click on Finish. After NetBeans has created the project, expand Project Files in the project tree and open the pom.xml file. In the <dependencies> tag, insert the following XML code: <dependencies> <dependency> <groupId>org.neo4j</groupId> <artifactId>neo4j</artifactId> <version>2.0.1</version> </dependency> </dependencies> <repositories> <repository> <id>neo4j</id> <url>http://m2.neo4j.org/content/repositories/releases/</url> <releases> <enabled>true</enabled> </releases> </repository> </repositories>   This code instructs Maven the dependency we are using on our project, that is, Neo4j. The version we have used here is 2.0.1. Of course, you can specify the latest available version. Once saved, the Maven file resolves the dependency, downloads the JAR files needed, and updates the Java build path. Now, the project is ready to use Neo4j and Cypher. Creating an embedded database Creating an embedded database is straightforward. First of all, to create a database, we need a GraphDatabaseFactory class, which can be done with the following code: GraphDatabaseFactory graphDbFactory = new GraphDatabaseFactory();   Then, we can invoke the newEmbeddedDatabase method with the following code: GraphDatabaseService graphDb = graphDbFactory .newEmbeddedDatabase("data/dbName");   Now, with the GraphDatabaseService class, we can fully interact with the database, create nodes, create relationships, set properties and indexes. Invoking Cypher from Java To execute Cypher queries on a Neo4j database, you need an instance of ExecutionEngine; this class is responsible for parsing and running Cypher queries, returning results in a ExecutionResult instance: import org.neo4j.cypher.javacompat.ExecutionEngine; import org.neo4j.cypher.javacompat.ExecutionResult; // ... ExecutionEngine engine = new ExecutionEngine(graphDb); ExecutionResult result = engine.execute("MATCH (e:Employee) RETURN e");   Note that we use the org.neo4j.cypher.javacompat package and not the org.neo4j.cypher package even though they are almost the same. The reason is that Cypher is written in Scala, and Cypher authors provide us with the former package for better Java compatibility. Now with the results, we can do one of the following options: Dumping to a string value Converting to a single column iterator Iterating over the full row Dumping to a string is useful for testing purposes: String dumped = result.dumpToString();   If we print the dumped string to the standard output stream, we will get the following result: Here, we have a single column (e) that contains the nodes. Each node is dumped with all its properties. The numbers between the square brackets are the node IDs, which are the long and unique values assigned by Neo4j on the creation of the node. When the result is single column, or we need only one column of our result, we can get an iterator over one column with the following code: import org.neo4j.graphdb.ResourceIterator; // ... ResourceIterator<Node> nodes = result.columnAs("e");   Then, we can iterate that column in the usual way, as shown in the following code: while(nodes.hasNext()) { Node node = nodes.next(); // do something with node }   However, Neo4j provides a syntax-sugar utility to shorten the code that is to be iterated: import org.neo4j.helpers.collection.IteratorUtil; // ... for (Node node : IteratorUtil.asIterable(nodes)) { // do something with node }   If we need to iterate over a multiple-column result, we would write this code in the following way: ResourceIterator<Map<String, Object>> rows = result.iterator(); for(Map<String,Object> row : IteratorUtil.asIterable(rows)) { Node n = (Node) row.get("e"); try(Transaction t = n.getGraphDatabase().beginTx()) { // do something with node } }   The iterator function returns an iterator of maps, where keys are the names of the columns. Note that when we have to work with nodes, even if they are returned by a Cypher query, we have to work in transaction. In fact, Neo4j requires that every time we work with the database, either reading or writing to the database, we must be in a transaction. The only exception is when we launch a Cypher query. If we launch the query within an existing transaction, Cypher will work as any other operation. No change will be persisted on the database until we commit the transaction, but if we run the query outside any transaction, Cypher will open a transaction for us and will commit changes at the end of the query. Summary We have now completed the setting up of a Neo4j database. We also learned about Cypher pattern matching. Resources for Article: Further resources on this subject: OpenSceneGraph: Advanced Scene Graph Components [Article] Creating Network Graphs with Gephi [Article] Building a bar graph cityscape [Article]

(For more resources related to this topic, see here.) Functional overview of Salesforce CRM The Salesforce CRM functions are related to each other and, as mentioned previously, have cross-over areas which can be represented as shown in the following diagram: Marketing administration Marketing administration is available in Salesforce CRM under the application suite known as the Marketing Cloud. The core functionality enables organizations to manage marketing campaigns from initiation to lead development in conjunction with the sales team. The features in the marketing application can help measure the effectiveness of each campaign by analyzing the leads and opportunities generated as a result of specific marketing activities. Salesforce automation Salesforce automation is the core feature set within Salesforce CRM and is used to manage the sales process and activities. It enables salespeople to automate manual and repetitive tasks and provides them with information related to existing and prospective customers. In Salesforce CRM, Salesforce automation is known as the Sales Cloud, and helps the sales people manage sales activities, leads and contact records, opportunities, quotes, and forecasts. Customer service and support automation Customer service and support automation within Salesforce CRM is known as the Service Cloud, and allows support teams to automate and manage the requests for service and support by existing customers. Using the Service Cloud features, organizations can handle customer requests, such as the return of faulty goods or repairs, complaints, or provide advice about products and services. Associated with the functional areas, described previously, are features and mechanisms to help users and customers collaborate and share information known as enterprise social networking. Enterprise social networking Enterprise social network capabilities within Salesforce CRM enable organizations to connect with people and securely share business information in real time. Social networking within an enterprise serves to connect both employees and customers and enables business collaboration. In Salesforce CRM, the enterprise social network suite is known as Salesforce Chatter. Salesforce CRM record life cycle The capabilities of Salesforce CRM provides for the processing of campaigns through to customer acquisition and beyond as shown in the following diagram: At the start of the process, it is the responsibility of the marketing team to develop suitable campaigns in order to generate leads. Campaign management is carried out using the Marketing Administration tools and has links to the lead and also any opportunities that have been influenced by the campaign. When validated, leads are converted to accounts, contacts, and opportunities. This can be the responsibility of either the marketing or sales teams and requires a suitable sales process to have been agreed upon. In Salesforce CRM, an account is the company or organization and a contact is an individual associated with an account. Opportunities can either be generated from lead conversion or may be entered directly by the sales team. As described earlier in this article, the structure of Salesforce requires account ownership to be established which sees inherited ownership of the opportunity. Account ownership is usually the responsibility of the sales team. Opportunities are worked through a sales process using sales stages where the stage is advanced to the point where they are set as won/closed and become sales. Opportunity information should be logged in the organization's financial system. Upon financial completion and acceptance of the deal (and perhaps delivery of the goods or service), the post-customer acquisition process is then enabled where the account and contact can be recognized as a customer. Here the customer relationships concerning incidents and requests are managed by escalating cases within the customer services and support automation suite.