Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
All Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

6719 Articles
article-image-developing-reports-using-rdp-and-report-contracts
Packt
07 Oct 2013
4 min read
Save for later

Developing Reports Using RDP and Report Contracts

Packt
07 Oct 2013
4 min read
(For more resources related to this topic, see here.) The Report data provider class A Report data provider class is commonly known as RDP. RDP is the data source type which is available when we add a new dataset to the report in Visual Studio. RDP is a class which resides inside AX and executes the business logic, processes the data, and returns a dataset which is rendered in the report. A Report data provider class should be ideally used in the following cases: We cannot directly use a query to access the data from the database The data has to prepare on the basis of business logic To define a Report data provider class, we use the following syntax: Sample RDP The Report contract class Report contracts in AX 2012 are used for defining parameters to the SSRS report. We can define any number of parameters using X++ statements of any data type, which can be passed on to the RDP class. And then, we can use the same contracts to query data from the database engine which will decrease an overhead on execution of a query in SQL. To define a Report contract class we use the following syntax: Report Contract Walkthrough – Creating an Auto Design report using the RDP class Scenario Matt, a Sales manager, needs a report to analyze total sales for customers. This walkthrough illustrates the following tasks: Creating a Report data provider class Creating a Report Model project Creating a Table report using Auto Design Saving to AOT, deploying, and running the report Prerequisites To learn and implement the following walkthrough, you must have: Microsoft Dynamics AX 2012 with sample data Microsoft Visual Studio 2010 with Microsoft Dynamics AX reporting extension Microsoft Dynamics AX 2012 SQL Server Reporting Services Creating a Report data provider class Open Microsoft Dynamics AX 2012 from the Start menu. Open the development workspace. You can do it in either of the following ways: Press Ctrl + D to open AOT in Development Workspace. Press Ctrl + Shift + P to open Projects in Development Workspace. Press Alt + W to open windows and select New Development Workspace. Press Ctrl + Shift + W to open New Development Workspace. Navigate to the AOT | Classes node. Right-click on the Classes node, and click on New | Class. Double-click on the class created and change the class declaration, as shown in the following code: class SrsRDPSample extends SRSReportDataProviderBase{ CustTransTotalSales custTransTotalSales; } Right-click on the SrsRDPSample class and select New | Method. Modify the method, as shown in the following code: [SRSReportDataSetAttribute('CustSales')] public CustTransTotalSales getTmpCustTable(){ return custTransTotalSales; } Right-click on the SrsRDPSample class and navigate to Override method | processReport. Modify the method, as shown in the following code: public void processReport(){ select * from custTransTotalSales; } Creating a Report Model project Start Visual Studio and press Ctrl + N to create a new project. Select Microsoft Dynamics AX under Installed Templates from the left pane, and select Report Model. Provide a name for the project as CustomerTotalSales_Autodesign. New project Creating a Table report using Auto Design Right-click on Solution, select Report under the Add submenu. Select the report and rename it to CustTotalSales_Autodesign. Right-click on Datasets and click on Add dataset. Modify the following properties for the newly added dataset: Data source to Microsoft Dynamics AX. Data source type to Query. Default Layout to Matrix. Name to CustTotalSalesDS. Switch to the Query property, and click on the button besides the name of the entity button to open the Query dialog. Select SrsRDPSample from the list and click on the Next button. New dataset: CustTotalSalesDS Select all fields from CustSales. Drag-and-drop CustTotalSalesDS to the Design section of the report. This will create a new Auto Design named as AutoDesign1. Select AutoDesign1, go to Properties, and set the following properties: LayoutTemplate to ReportLayoutStyleTemplate Title to Customer sales. Select CustTotalSalesDS under AutoDesign1, and set the following properties: StyleTemplate to TableStyleAlternatingRowsTemplate. Title to Customer sales. Drag the AccountNum field from the Data node to Groupings. Drag the TransDate field to the Sorting node. Under the Data node, select only the TransDate, AmountMST, CurrencyCode, DocumentDate, DueDate, LastSettleDate, PaymMode, and TransType1 fields. Report design Saving to AOT, deploying, and running the report Save the report to AOT and Deploy to the Report Server, and then run the report. Summary In this article we covered Report data provider (RDP) class and Report contract class. We also learned how to implement these classes for our reporting needs. We learned creating an Auto Design report using the RDP class by following the step-by-step walkthrough. Resources for Article: Further resources on this subject: Testing Workflows for Microsoft Dynamics AX 2009 Administration [Article] Types of services in Microsoft Dynamics AX 2012 [Article] Installing the Dynamics AX Base Server Components for Microsoft [Article]
Read more
  • 0
  • 0
  • 6147

article-image-building-your-first-zend-framework-application
Packt
26 Jul 2013
15 min read
Save for later

Building Your First Zend Framework Application

Packt
26 Jul 2013
15 min read
(For more resources related to this topic, see here.) Prerequisites Before you get started with setting up your first ZF2 Project, make sure that you have the following software installed and configured in your development environment: PHP Command Line Interface Git : Git is needed to check out source code from various github.com repositories Composer : Composer is the dependency management tool used for managing PHP dependencies The following commands will be useful for installing the necessary tools to setup a ZF2 Project: To install PHP Command Line Interface: $ sudo apt-get install php5-cli To install Git: $ sudo apt-get install git To install Composer: $ curl -s https://getcomposer.org/installer | php ZendSkeletonApplication ZendSkeletonApplication provides a sample skeleton application that can be used by developers as a starting point to get started with Zend Framework 2.0. The skeleton application makes use of ZF2 MVC, including a new module system. ZendSkeletonApplication can be downloaded from GitHub (https://github.com/zendframework/ZendSkeletonApplication). Time for action – creating a Zend Framework project To set up a new Zend Framework project, we will need to download the latest version of ZendSkeletonApplication and set up a virtual host to point to the newly created Zend Framework project. The steps are given as follows: Navigate to a folder location where you want to set up the new Zend Framework project: $ cd /var/www/ Clone the ZendSkeletonApplication app from GitHub: $ git clone git://github.com/zendframework/ ZendSkeletonApplication.git CommunicationApp In some Linux configurations, necessary permissions may not be available to the current user for writing to /var/www. In such cases, you can use any folder that is writable and make necessary changes to the virtual host configuration. Install dependencies using Composer: $ cd CommunicationApp/ $ php composer.phar self-update $ php composer.phar install The following screenshot shows how Composer downloads and installs the necessary dependencies: Before adding a virtual host entry we need to set up a hostname entry in our hosts file so that the system points to the local machine whenever the new hostname is used. In Linux this can be done by adding an entry to the /etc/hosts file: $ sudo vim /etc/hosts In Windows, this file can be accessed at %SystemRoot%system32driversetchosts. Add the following line to the hosts file: 127.0.0.1 comm-app.local The final hosts file should look like the following: Our next step would be to add a virtual host entry on our web server; this can be done by creating a new virtual host's configuration file: $ sudo vim /usr/local/zend/etc/sites.d/vhost_comm-app-80.conf This new virtual host filename could be different for you depending upon the web server that you use; please check out your web server documentation for setting up new virtual hosts. For example, if you have Apache2 running on Linux, you will need to create the new virtual host file in /etc/apache2/sites-available and enable the site using the command a2ensite comm-app.local. Add the following configuration to the virtual host file: <VirtualHost *:80> ServerName comm-app.local DocumentRoot /var/www/CommunicationApp/public SetEnv APPLICATION_ENV "development" <Directory /var/www/CommunicationApp/public> DirectoryIndex index.php AllowOverride All Order allow,deny Allow from all </Directory> </VirtualHost> If you are using a different path for checking out the ZendSkeletonApplication project make sure that you include that path for both DocumentRoot and Directory directives. After configuring the virtual host file, the web server needs to be restarted: $ sudo service zend-server restart Once the installation is completed, you should be able to open http://comm-app.local on your web browser. This should take you to the following test page : Test rewrite rules In some cases, mod_rewrite may not have been enabled in your web server by default; to check if the URL redirects are working properly, try to navigate to an invalid URL such as http://comm-app.local/12345; if you get an Apache 404 page, then the .htaccess rewrite rules are not working; they will need to be fixed, otherwise if you get a page like the following one, you can be sure of the URL working as expected. What just happened? We have successfully created a new ZF2 project by checking out ZendSkeletonApplication from GitHub and have used Composer to download the necessary dependencies including Zend Framework 2.0. We have also created a virtual host configuration that points to the project's public folder and tested the project in a web browser. Alternate installation options We have seen just one of the methods of installing ZendSkeletonApplication; there are other ways of doing this. You can use Composer to directly download the skeleton application and create the project using the following command: $ php composer.phar create-project --repositoryurl="http://packages.zendframework.com" zendframework/skeleton-application path/to/install You can also use a recursive Git clone to create the same project: $ git clone git://github.com/zendframework/ZendSkeletonApplication.git --recursive Refer to: http://framework.zend.com/downloads/skeleton-app Zend Framework 2.0 – modules In Zend Framework, a module can be defined as a unit of software that is portable and reusable and can be interconnected to other modules to construct a larger, complex application. Modules are not new in Zend Framework, but with ZF2, there is a complete overhaul in the way modules are used in Zend Framework. With ZF2, modules can be shared across various systems, and they can be repackaged and distributed with relative ease. One of the other major changes coming into ZF2 is that even the main application is now converted into a module; that is, the application module. Some of the key advantages of Zend Framework 2.0 modules are listed as follows: Self-contained, portable, reusable Dependency management Lightweight and fast Support for Phar packaging and Pyrus distribution Zend Framework 2.0 – project folder structure The folder layout of a ZF2 project is shown as follows: Folder name Description config Used for managing application configuration. data Used as a temporary storage location for storing application data including cache files, session files, logs, and indexes. module Used to manage all application code. module/Application This is the default application module that is provided with ZendSkeletonApplication. public This is the default application module that is provided with ZendSkeletonApplication. vendor Used to manage common libraries that are used by the application. Zend Framework is also installed in this folder. vendor/zendframework endor/zendframework Zend Framework 2.0 is installed here. Time for action – creating a module Our next activity will be about creating a new Users module in Zend Framework 2.0. The Users module will be used for managing users including user registration, authentication, and so on. We will be making use of ZendSkeletonModule provided by Zend, shown as follows: Navigate to the application's module folder: $ cd /var/www/CommunicationApp/ $ cd module/ Clone ZendSkeletonModule into a desired module name, in this case it is Users: $ git clone git://github.com/zendframework/ZendSkeletonModule.git Users After the checkout is complete, the folder structure should look like the following screenshot: Edit Module.php ; this file will be located in the Users folder under modules (CommunicationApp/module/Users/module.php) and change the namespace to Users. Replace namespace ZendSkeletonModule; with namespace Users;. The following folders can be removed because we will not be using them in our project: * Users/src/ZendSkeletonModule * Users/view/zend-skeleton-module What just happened? We have installed a skeleton module for Zend Framework; this is just an empty module, and we will need to extend this by creating custom controllers and views. In our next activity, we will focus on creating new controllers and views for this module. Creating a module using ZFTool ZFTool is a utility for managing Zend Framework applications/projects, and it can also be used for creating new modules; in order to do that, you will need to install ZFTool and use the create module command to create the module using ZFTool: $ php composer.phar require zendframework/zftool:dev-master $ cd vendor/zendframework/zftool/ $ php zf.php create module Users2 /var/www/CommunicationApp Read more about ZFTool at the following link: http://framework.zend.com/manual/2.0/en/modules/zendtool.introduction.html MVC layer The fundamental goal of any MVC Framework is to enable easier segregation of three layers of the MVC, namely, model, view, and controller. Before we get to the details of creating modules, let's quickly try to understand how these three layers work in an MVC Framework: Model : The model is a representation of data; the model also holds the business logic for various application transactions. View : The view contains the display logic that is used to display the various user interface elements in the web browser. Controller : The controller controls the application logic in any MVC application; all actions and events are handled at the controller layer. The controller layer serves as a communication interface between the model and the view by controlling the model state and also by representing the changes to the view. The controller also provides an entry point for accessing the application. In the new ZF2 MVC structure, all the models, views, and controllers are grouped by modules. Each module will have its own set of models, views, and controllers, and will share some components with other modules. Zend Framework module – folder structure The folder structure of Zend Framework 2.0 module has three vital components—the configurations, the module logic, and the views. The following table describes how contents in a module are organized: Folder name Description config Used for managing module configuration src Contains all module source code, including all controllers and models view Used to store all the views used in the module Time for action – creating controllers and views Now that we have created the module, our next step would be having our own controllers and views defined. In this section, we will create two simple views and will write a controller to switch between them: Navigate to the module location: $ cd /var/www/CommunicationApp/module/Users Create the folder for controllers: $ mkdir -p src/Users/Controller/ Create a new IndexController file, < ModuleName >/src/<ModuleName>/Controller/: $ cd src/Users/Controller/ $ vim IndexController.php Add the following code to the IndexController file: <?php namespace UsersController; use ZendMvcControllerAbstractActionController; use ZendViewModelViewModel; class IndexController extends AbstractActionController { public function indexAction() { $view = new ViewModel(); return $view; } public function registerAction() { $view = new ViewModel(); $view->setTemplate('users/index/new-user'); return $view; } public function loginAction() { $view = new ViewModel(); $view->setTemplate('users/index/login'); return $view; } } The preceding code will do the following actions; if the user visits the home page, the user is shown the default view; if the user arrives with an action register, the user is shown the new-user template; and if the user arrives with an action set to login, then the login template is rendered. Now that we have created the controller, we will have to create necessary views to render for each of the controller actions. Create the folder for views: $ cd /var/www/CommunicationApp/module/Users $ mkdir -p view/users/index/ Navigate to the views folder, <Module>/view/<module-name>/index: $ cd view/users/index/ Create the following view files: index login new-user For creating the view/users/index/index.phtml file, use the following code: <h1>Welcome to Users Module</h1> <a href="/users/index/login">Login</a> | <a href = "/users/index/register">New User Registration</a> For creating the view/users/index/login.phtml file, use the following code: <h2> Login </h2> <p> This page will hold the content for the login form </p> <a href="/users"><< Back to Home</a> For creating the view/users/index/new-user.phtml file, use the following code: <h2> New User Registration </h2> <p> This page will hold the content for the registration form </p> <a href="/users"><< Back to Home</a> What just happened? We have now created a new controller and views for our new Zend Framework module; the module is still not in a shape to be tested. To make the module fully functional we will need to make changes to the module's configuration, and also enable the module in the application's configuration. Zend Framework module – configuration Zend Framework 2.0 module configuration is spread across a series of files which can be found in the skeleton module. Some of the configuration files are described as follows: Module.php: The Zend Framework 2 module manager looks for the Module.php file in the module's root folder. The module manager uses the Module.php file to configure the module and invokes the getAutoloaderConfig() and getConfig() methods. autoload_classmap.php: The getAutoloaderConfig() method in the skeleton module loads autoload_classmap.php to include any custom overrides other than the classes loaded using the standard autoloader format. Entries can be added or removed to the autoload_classmap.php file to manage these custom overrides. config/module.config.php: The getConfig() method loads config/module.config.php; this file is used for configuring various module configuration options including routes, controllers, layouts, and various other configurations. Time for action – modifying module configuration In this section will make configuration changes to the Users module to enable it to work with the newly created controller and views using the following steps: Autoloader configuration – The default autoloader configuration provided by the ZendSkeletonModule needs to be disabled; this can be done by editing autoload_classmap.php and replacing it with the following content: <?php return array(); Module configuration – The module configuration file can be found in config/module.config.php; this file needs to be updated to reflect the new controllers and views that have been created, as follows: Controllers – The default controller mapping points to the ZendSkeletonModule; this needs to be replaced with the mapping shown in the following snippet: 'controllers' => array( 'invokables' => array( 'UsersControllerIndex' => 'UsersControllerIndexController', ), ), Views – The views for the module have to be mapped to the appropriate view location. Make sure that the view uses lowercase names separated by a hyphen (for example, ZendSkeleton will be referred to as zend-skeleton): 'view_manager' => array( 'template_path_stack' => array( 'users' => __DIR__ . '/../view', ), ), Routes – The last module configuration is to define a route for accessing this module from the browser; in this case we are defining the route as /users, which will point to the index action in the Index controller of the Users module: 'router' => array( 'routes' => array( 'users' => array( 'type' => 'Literal', 'options' => array( 'route' => '/users', 'defaults' => array( '__NAMESPACE__' => 'UsersController', 'controller' => 'Index', 'action' => 'index', ), ), After making all the configuration changes as detailed in the previous sections, the final configuration file, config/module.config.php, should look like the following: <?php return array( 'controllers' => array( 'invokables' => array( 'UsersControllerIndex' => 'UsersControllerIndexController', ), ), 'router' => array( 'routes' => array( 'users' => array( 'type' => 'Literal', 'options' => array( // Change this to something specific to your module 'route' => '/users', 'defaults' => array( //Change this value to reflect the namespace in which // the controllers for your module are found '__NAMESPACE__' => 'UsersController', 'controller' => 'Index', 'action' => 'index', ), ), 'may_terminate' => true, 'child_routes' => array( // This route is a sane default when developing a module; // as you solidify the routes for your module, however, // you may want to remove it and replace it with more // specific routes. 'default' => array( 'type' => 'Segment', 'options' => array( 'route' => '/[:controller[/:action]]', 'constraints' => array( 'controller' => '[a-zA-Z][a-zA-Z0-9_-]*', 'action' => '[a-zA-Z][a-zA-Z0-9_-]*', ), 'defaults' => array( ), ), ), ), ), ), ), 'view_manager' => array( 'template_path_stack' => array( 'users' => __DIR__ . '/../view', ), ), ); Application configuration – Enable the module in the application's configuration—this can be done by modifying the application's config/application.config.php file, and adding Users to the list of enabled modules: 'modules' => array( 'Application', 'Users', ), To test the module in a web browser, open http://comm-app.local/users/ in your web browser; you should be able to navigate within the module. The module home page is shown as follows: The registration page is shown as follows: What just happened? We have modified the configuration of ZendSkeletonModule to work with the new controller and views created for the Users module. Now we have a fully-functional module up and running using the new ZF module system. Have a go hero Now that we have the knowledge to create and configure own modules, your next task would be to set up a new CurrentTime module. The requirement for this module is to render the current time and date in the following format: Time: 14:00:00 GMT Date: 12-Oct-2012 Summary We have now learned about setting up a new Zend Framework project using Zend's skeleton application and module. In our next chapters, we will be focusing on further development on this module and extending it into a fully-fledged application. Resources for Article : Further resources on this subject: Magento's Architecture: Part 2 [Article] Authentication with Zend_Auth in Zend Framework 1.8 [Article] Authorization with Zend_Acl in Zend Framework 1.8 [Article]
Read more
  • 0
  • 0
  • 6143

article-image-aspnet-site-performance-reducing-long-wait-times
Packt
12 Oct 2010
8 min read
Save for later

ASP.NET site performance: reducing long wait times

Packt
12 Oct 2010
8 min read
Measuring wait times We can use a number of ways to find out which external requests are most frequent and how long the site has to wait for a response: Run the code in the debugger with breakpoints around each external request. This will give you a quick hint of which external request is the likely culprit. However, you wouldn't do this in a production environment, as it only gives you information for a few requests. Use the Trace class (in the namespace System.Diagnostics) to trace how long each request takes. This will give you a lot of detailed information. However, the overhead incurred by processing all the trace messages may be too high to use in a production environment, and you would have to somehow aggregate the trace data to find which requests are the most frequent and take the longest. Build performance counters into your code that record the frequency of each request and the average wait time. These counters are light-weight, and hence, can be used in a production environment. Also, you can readily access them via perfmon, along with the counters provided by ASP.NET, SQL Server, and so on that you have already come across. The remainder of this section focuses on performance counters. Also, performance counters are a convenient way to keep an eye on off-box requests on a day-to-day basis instead of as a one-off. Windows offers you 28 types of performance counters to choose from. Some of these are esoteric, others extremely useful. For example, you can measure the rate per second that a request is made, and the average time in milliseconds that the site waits for a response. Adding your own custom counters is easy, and you can see their real-time values in perfmon, along with that of the built-in counters. The runtime overhead of counters is minimal. You have already come across some of the hundreds of counters published by ASP.NET, SQL Server, and Windows itself. Even if you add a lot of counters, CPU overhead would be well under one percent. This section describes only three commonly used counters: simple number, rate per second, and time. A list of all types of counters with examples of their use is available at http://msdn.microsoft.com/en-us/library/system.diagnostics.performancecountertype.aspx?ppud=4. To use the counters, you need to follow these three steps: Create custom counters. Update them in your code. See their values in perfmon. Creating custom counters In this example, we'll put counters on a page that simply waits for one second to simulate waiting for an external resource. Windows allows you to group counters into categories. We'll create a new category "Test Counters" for the new counters. Counter NameCounter TypeDescriptionNbr Page HitsNumberOfItems6464 bit counter, counting the total number of hits on the page since the website started.Hits/secondRateOfCountsPerSecond32Hits per secondAverage WaitAverageTimer32Time taken by the resource. Inspite of the name, it is used here to simply measure an interval, not an average.Average Wait Base*AverageBaseUtility counter required by Average Wait. *The text says there are three counters, but the table lists four. Why? The last counter, Average Wait Base, doesn't provide information on its own, but helps to compute the value of counter Average Wait. Later on, we'll see how this works. There are two ways to create the "Test Counters" category and the counters themselves: Using Visual Studio: This is relatively quick, but if you want to apply the same counters to for example your development and production environments, you'll have to enter the counters separately in each environment Programmatically: Because this involves writing code, it takes a bit longer upfront, but makes it easier to apply the same counters to multiple environments and to place the counters under source control Creating counters with Visual Studio To create the counters in Visual Studio: Make sure you have administrative privileges or are a member of the Performance Monitor Users group. Open Visual Studio. Click on the Server Explorer tab. Expand Servers. Expand your machine. Right-click on Performance Counters and choose Create New Category. Enter Test Counters in the Category Name field. Click on the New button for each of the four counters to add, as listed in the table you saw earlier. Be sure to add the Average Wait Base counter right after Average Wait, to properly associate the two counters. Click on OK when you're done. This technique is easy. However, you'll need to remember to add the same counters to the production machine when you release new code with new custom counters. Writing a program to create the counters is more work initially, but gives you easier maintenance in the long run. Let's see how to do this. Creating counters programmatically From a maintenance point of view, it would be best to create the counters when the web application starts, in the Global.asax file. However, you would then have to make the account under which the application pool runs part of the Performance Monitor Users group. An alternative is to create the counters in a separate console program. An administrator can then run the program to create the counters on the server. Here is the code. using System; using System.Diagnostics; namespace CreateCounters { class Program { static void Main(string[] args) { To create a group of counters, you create each one in turn, and add them to a CounterCreationDataCollection object: CounterCreationDataCollection ccdc = new CounterCreationDataCollection(); Create the first counter, Nbr Page Hits. Give it a short help message and the counter type. Now, add it to the CounterCreationDataCollection object: CounterCreationData ccd = new CounterCreationData ("Nbr Page Hits", "Total number of page hits", PerformanceCounterType.NumberOfItems64); ccdc.Add(ccd); Add the second, third, and fourth counters along the same lines: ccd = new CounterCreationData("Hits / second", "Total number of page hits / sec", PerformanceCounterType.RateOfCountsPerSecond32); ccdc.Add(ccd); ccd = new CounterCreationData("Average Wait", "Average wait in seconds", PerformanceCounterType.AverageTimer32); ccdc.Add(ccd); ccd = new CounterCreationData("Average Wait Base", "", PerformanceCounterType.AverageBase); ccdc.Add(ccd); Now, it's time to take the CounterCreationDataCollection object and make it into a category. Because you'll get an exception when you try to create a category that already exists if there already is a category with the same name, delete it now. Because you can't add new counters to an existing category, there is no simple work-around for this: if (PerformanceCounterCategory.Exists("Test Counters")) { PerformanceCounterCategory.Delete("Test Counters"); } Finally, create the Test Counters category. Give it a short help message, and make it a single instance. You can also make a category multi-instance, which allows you to split the category into instances. Also, pass in the CounterCreationDataCollection object with all the counters. This creates the complete category with all your counters in one go, as shown in the following code: PerformanceCounterCategory.Create("Test Counters", "Counters for test site",PerformanceCounterCategoryType. SingleInstance,ccdc); } } } Now that you know how to create the counters, let's see how to update them in your code Updating counters in your code To keep things simple, this example uses the counters in a page that simply waits for a second to simulate waiting for an external resource: using System; using System.Diagnostics; public partial class _Default : System.Web.UI.Page { protected void Page_Load(object sender, EventArgs e) { First, increment the nbrPageHits counter. To do this, create a PerformanceCounter object, attaching it to the nbrPageHits counter in the Test Counters category. Then, increment the PerformanceCounter object: PerformanceCounter nbrPageHitsCounter = new PerformanceCounter("Test Counters", "Nbr Page Hits", false); nbrPageHitsCounter.Increment(); Now, do the same with the Hits/second counter. Because you set its type to RateOfCountsPerSecond32 when you generated it in the console program, the counter will automatically give you a rate per second when viewed in perfmon: PerformanceCounter nbrPageHitsPerSecCounter = new PerformanceCounter("Test Counters", "Hits / second", false); nbrPageHitsPerSecCounter.Increment(); To measure how long the actual operation takes, create a Stopwatch object, and start it: Stopwatch sw = new Stopwatch(); sw.Start(); Execute the simulated operation: // Simulate actual operation System.Threading.Thread.Sleep(1000); Stop the stopwatch: sw.Stop(); Update the Average Wait counter and the associated Average Wait Base counter to record the elapsed time in the stopwatch. PerformanceCounter waitTimeCounter = new PerformanceCounter("Test Counters", "Average Wait", false); waitTimeCounter.IncrementBy(sw.ElapsedTicks); PerformanceCounter waitTimeBaseCounter = new PerformanceCounter("Test Counters", "Average Wait Base", false); waitTimeBaseCounter.Increment(); } } Now that we've seen how to create and use the most commonly used counters, it's time to retrieve their values. Viewing custom counters in perfmon Accessing your custom counters goes the following way: On the server, run perfmon from the command prompt. To open the command prompt on Vista, click on Start | All Programs | Accessories | Command Prompt. This opens the monitor window. Expand Monitoring Tools and click on Performance Monitor. Click on the green "plus" sign. In the Add Counters dialog, scroll down to your new Test Counters category. Expand that category and add your new counters. Click on OK. To see the counters in action, run a load test. If you use WCAT, you could use files runwcat_testcounters.bat and testcounters_scenario.ubr from the downloaded code bundle.
Read more
  • 0
  • 0
  • 6138

article-image-exploring-advanced-interactions-webdriver
Packt
21 Jan 2014
9 min read
Save for later

Exploring Advanced Interactions of WebDriver

Packt
21 Jan 2014
9 min read
(For more resources related to this topic, see here.) Understanding actions, build, and perform We know how to take some basic actions, such as clicking on a button and typing text into a textbox; however, there are many scenarios where we have to perform multiple actions at the same time. For example, keeping the Shift button pressed and typing text for uppercase letters, and the dragging and dropping mouse actions. Let's see a simple scenario here. Open the selectable.html file that is attached with this book. You will see tiles of numbers from 1 to 12. If you inspect the elements with Firebug, you will see an ordered list tag (<ol>) and 12 list items (<li>) under it, as shown in the following code: <ol id="selectable" class="ui-selectable"> <li class="ui-state-default ui-selectee" name="one">1</li> <li class="ui-state-default ui-selectee" name="two">2</li> <li class="ui-state-default ui-selectee" name="three">3</li> <li class="ui-state-default ui-selectee" name="four">4</li> <li class="ui-state-default ui-selectee" name="five">5</li> <li class="ui-state-default ui-selectee" name="six">6</li> <li class="ui-state-default ui-selectee" name="seven">7</li> <li class="ui-state-default ui-selectee" name="eight">8</li> <li class="ui-state-default ui-selectee" name="nine">9</li> <li class="ui-state-default ui-selectee" name="ten">10</li> <li class="ui-state-default ui-selectee" name="eleven">11</li> <li class="ui-state-default ui-selectee" name="twelve">12</li> </ol> If you click a number, it's background color changes to orange. Try selecting the 1, 3, and 5 numbered tiles. You do that by holding the Ctrl key + 1 numbered tile + 3 numbered tile + 5 numbered tile. So, this involves performing multiple actions, that is, holding the Ctrl key continuously and clicking on 1, 3, and 5 tiles. How do we perform these multiple actions using WebDriver? The following code demonstrates that: public class ActionBuildPerform {     public static void main(String... args) {       WebDriver driver = new FirefoxDriver();       driver.get("file://C:/selectable.html");       WebElement one = driver.findElement(By.name("one"));       WebElement three = driver.findElement(By.name("three"));      WebElement five = driver.findElement(By.name("five"));       // Add all the actions into the Actions builder.      Actions builder = new Actions(driver);         builder.keyDown( Keys.CONTROL )               .click(one)              .click(three)              .click(five)              .keyUp(Keys.CONTROL);        // Generate the composite action.        Action compositeAction = builder.build();        // Perform the composite action.        compositeAction.perform( );       }    } Now, if you see the code, line number 9 is where we are getting introduced to a new class named Actions. This Actions class is the one that is used to emulate all the complex user events. Using this, the developer of the test script could combine all the necessary user gestures into one composite action. From line 9 to line 14, we have declared all the actions that are to be executed to achieve the functionality of clicking on the numbers 1, 3, and 5. Once all the actions are grouped together, we build that into a composite action. This is contained on line 16. Action is an interface that has only the perform() method, which executes the composite action. Line 18 is where we are actually executing the action using the perform() method. So, to make WebDriver perform multiple actions at the same time, you need to follow a three-step process of using the user-facing API of the Actions class to group all the actions, then build the composite action, and then the perform the action. This process can be made into a two-step process as the perform() method internally calls the build() method. So the previous code will look as follows: public class ActionBuildPerform {     public static void main(String... args) {       WebDriver driver = new FirefoxDriver();       driver.get("file://C:/selectable.html");       WebElement one = driver.findElement(By.name("one"));       WebElement three = driver.findElement(By.name("three"));      WebElement five = driver.findElement(By.name("five"));       // Add all the actions into the Actions builder.     Actions builder = new Actions(driver);         builder.keyDown( Keys.CONTROL )               .click(one)              .click(three)              .click(five)              .keyUp(Keys.CONTROL);        // Perform the action.        builder.perform( );   } } In the preceding code, we have directly invoked the perform() method on the Actions instance, which internally calls the build() method to create a composite action before executing it. In the subsequent sections of this article, we will take a closer look at the Actions class. All the actions are basically divided into two categories: mouse-based actions and keyboard-based actions. In the following sections, we will discuss all the actions that are specific to the mouse and keyboard available in the Actions class. Learning mouse-based interactions There are around eight different mouse actions that can be performed using the Actions class. We will see each of their syntax and a working example. The moveByOffset action The moveByOffset method is used to move the mouse from its current position to another point on the web page. Developers can specify the X distance and Y distance the mouse has to be moved. When the page is loaded, generally the initial position of a mouse would be (0, 0), unless there is an explicit focus declared by the page. The API syntax for the moveByOffset method is as follows: public Actions moveByOffset(int xOffSet, int yOffSet) In the preceding code, xOffSet is the input parameter providing the WebDriver the amount of offset to be moved along the x axis. A positive value is used to move the cursor to the right, and a negative value is used to move the cursor to the left. yOffSet is the input parameter providing the WebDriver the amount of offset to be moved along the y axis. A positive value is used to move the cursor down along the y axis and a negative value is used to move the cursor toward the top. When the xOffSet and yOffSet values result in moving the cursor out of the document, a MoveTargetOutOfBoundsException is raised. Let's see a working example of it. The objective of the following code is to move the cursor on to the number 3 tile on the web page:  public class MoveByOffSet{   public static void main(String... args) {     WebDriver driver = new FirefoxDriver();     driver.get("file://C:/Selectable.html");     WebElement three = driver.findElement(By.name("three"));     System.out.println("X coordinate: "+three.getLocation().getX()+" Y coordinate: "+three.getLocation().getY());     Actions builder = new Actions(driver);     builder.moveByOffset(three.getLocation().getX()+1, three.getLocation().getY()+1);     builder.perform();   }  } We have added +1 to the coordinates, because if you observe the element in Firebug, we have a style border of 1 px. Border is a CSS-style attribute, which when applied to an element, will add a border of the specified color around the element with the specified amount of thickness. Though the previous code does move your mouse over tile 3, we don't realize it because we are not doing any action there. We will see that when we use this moveByOffset() method in combination with the click method shortly. The moveByOffset() method may not work in Mac OSX and may raise a JavaScript error when used independently like the previous code. The click at current location action The click method is used to simulate the left-click of your mouse at its current point of location. This method doesn't really realize where or on which element it is clicking. It just blindly clicks wherever it is at that point of time. Hence, this method is used in combination with some other action rather than independently, to create a composite action. The API syntax for the click method is as follows: public Actions click() The click method doesn't really have any context about where it is performing its action; hence, it doesn't take any input parameter. Let's see a code example of the click method: public class MoveByOffsetAndClick{   public static void main(String... args) {     WebDriver driver = new FirefoxDriver();     driver.get("file://C:/Selectable.html");     WebElement seven = driver.findElement(By.name("seven"));     System.out.println("X coordinate: "+seven.getLocation().getX()+" Y coordinate: "+seven.getLocation().getY());     Actions builder = new Actions(driver);     builder.moveByOffset( seven.getLocation ().getX()+1, seven.getLocation().getY()+1).click();     builder.perform();   } } Line 8 is where we have used a combination of the moveByOffset() and click() methods to move the cursor from point (0, 0) to the point of tile 7. Because the initial position of the mouse is (0, 0), the X, Y offset provided for the moveByOffset() method is nothing but the location of the tile 7 element. Now, lets try to move the cursor from tile 1 to tile 11 and from there to tile 5 and see how the code looks. Before we get into the code, let's inspect the selectable.html page using Firebug. The following is the style of each tile: #selectable li {     float: left;     font-size: 4em;     height : 80px;     text-align: center;     width : 100px; } .ui-state-default, .ui-widget-content .ui-state-default, .ui-widget-header .ui-state-default {     background: url("images/ui-bg_glass_75_e6e6e6_1x400.png") repeat-x scroll 50% 50% #E6E6E6;     border : 1px solid #D3D3D3;     color: #555555;     font-weight: normal; } The three elements with which we are concerned for our offset movement in the preceding style code are: height, width, and the border thickness. Here, the height value is 80px, width value is 100px, and border value is 1px. Use these three factors to calculate the offset to navigate from one tile to the other. Note that the border thickness between any two tiles will result in 2 px; that is, 1 px from each tile. The following is the code that uses the moveByOffset and click() methods to navigate from tile 1 to tile 11, and from there to tile 5: public class MoveByOffsetAndClick{   public static void main(String... args) {     WebDriver driver = new FirefoxDriver();     driver.get("file://C:/Selectable.html");     WebElement one = driver.findElement(By.name("one"));     WebElement eleven = driver.findElement(By.name("eleven"));     WebElement five = driver.findElement(By.name("five"));     int border = 1;     int tileWidth = 100;     int tileHeight = 80;     Actions builder = new Actions(driver);     //Click on One     builder.moveByOffset( one.getLocation ().getX()+border, one.getLocation().getY()+border).click();     builder.build().perform();     // Click on Eleven     builder.moveByOffset( 2*tileWidth+4*border, 2*tileHeight+4*border).click();     builder.build().perform();    //Click on Five     builder.moveByOffset( -2*tileWidth-4*border, -tileHeight-2*border).click();     builder.build().perform();    }  }
Read more
  • 0
  • 0
  • 6137

article-image-implementing-dependency-injection-in-swift-tutorial
Bhagyashree R
11 Feb 2019
14 min read
Save for later

Implementing Dependency Injection in Swift [Tutorial]

Bhagyashree R
11 Feb 2019
14 min read
In software development, it's always recommended to split the system into loosely coupled modules that can work independently as much as they can. Dependency Injection (DI) is a pattern that helps to reach this goal, creating a maintainable and testable system. It is often confused with complex and over-configurable frameworks that permit us to add DI to our code; in reality, it is a simple pattern that can be added without too much effort. This article is taken from the book Hands-On Design Patterns with Swift by Florent Vilmart, Giordano Scalzo, and Sergio De Simone.  This book demonstrates how to apply design patterns and best practices in real-life situations, whether that's for new or already existing Swift projects. You’ll begin with a quick refresher on Swift, the compiler, the standard library, and the foundation, followed by the Cocoa design patterns to follow up with the creational, structural, and behavioral patterns as defined by the GoF.  To follow along with the examples implemented in this article, you can download the code from the book’s GitHub repository. In this article, we'll see what Dependency Injection is, where it comes, and how it's defined so that we can then discuss various methods to implement it, having a clear understanding of its principles. Dependency Injection, a primer Dependency Injection is one of the most misunderstood concepts in computer programming. This is because the Dependency Injection borders are quite blurry and they could overlap with other object-oriented programming concepts. Let's start with a formal definition given by Wikipedia: "In software engineering, Dependency Injection is a software design pattern that implements inversion of control for resolving dependencies." To be honest, this is not really clear: what is Inversion of Control? Why is it useful for resolving dependencies? In procedural programming, each object interacts with all of its collaborators in a direct way and also instantiates them directly. In Inversion Of Control, this flow is managed by a third party, usually, a framework that calls the objects and receives notifications. An example of this is an implementation of a UI engine. In a UI Engine, there are two parts: the Views and the Models part. The Views part handles all the interaction with the users, such as tapping buttons and rendering labels, whereas the Models part is responsible for business logic. Usually, the application code goes in the Models part, and the connections with the Views are done via callbacks that are called by the engine when the user interacts with a button or a text field. The paradigm changes from an imperative style where the algorithm is a sequence of actions, like in do this then do that, to an event style, when the button is tapped then call the server. The control of the actions is thus inverted. Instead of being the model that does things, the model now receives calls. Inversion of Control is often called Hollywood Principle. The essence of this principle is, "Don't call us, we'll call you," which is a response you might hear after auditioning for a role in Hollywood. In procedural programming, the flow of the program is determined by the modules that are statically connected together: ContactsView talks to ContactsCoreData and  ContactsProductionRemoteService, and each object instantiate its next collaborator. In Inversion of Control, ContactsView talks to a generic ContactsStore and a generic ContactsRemoteService whose concrete implementation could change depending on the context. If it is during the tests, an important role is played by the entity that manages how to create and connect all the objects together. After having defined the concept of IoC, let's give a simpler definition of DI by James Shore: "Dependency Injection" is a 25-dollar term for a 5-cent concept. [...] Dependency Injection means giving an object its instance variables. Really. That's it." The first principle of the book Design Patterns by the Gang of Four is "Program to an interface, not an implementation" which means that the objects need to know each other only by their interface and not by their implementation. After having defined how all the classes in software will collaborate with each other, this collaboration can be designed as a graph. The graph could be implemented connecting together the actual implementation of the classes, but following the first principle mentioned previously, we can do it using the interfaces of the same objects: the Dependency Injection is a way of building this graph passing the concrete classes to the objects. Four ways to use Dependency Injection Dependency Injection is used ubiquitously in Cocoa too, and in the following examples, we'll see code snippets both from Cocoa and typical client-side code. Let's take a look at the following four sections to learn how to use Dependency Injection. Constructor Injection The first way to do DI is to pass the collaborators in the constructor, where they are then saved in private properties. Let's have as an example on e-commerce app, whose Basket is handled both locally and remotely. The BasketClient class orchestrates the logic, saves locally in BasketStore, and synchronizes remotely with BasketService: protocol BasketStore { func loadAllProduct() -> [Product] func add(product: Product) func delete(product: Product) } protocol BasketService { func fetchAllProduct(onSuccess: ([Product]) -> Void) func append(product: Product) func remove(product: Product) } struct Product { let id: String let name: String //... } Then in the constructor of BasketClient, the concrete implementations of the protocols are passed: class BasketClient { private let service: BasketService private let store: BasketStore init(service: BasketService, store: BasketStore) { self.service = service self.store = store } func add(product: Product) { store.add(product: product) service.append(product: product) calculateAppliedDiscount() //... } // ... private func calculateAppliedDiscount() { // ... } } In Cocoa and Cocoa Touch, the Apple foundation libraries, there are a few examples of this pattern. A notable example is NSPersistentStore in CoreData: class NSPersistentStore: NSObject { init(persistentStoreCoordinator root: NSPersistentStoreCoordinator?, configurationName name: String?, URL url: NSURL, options: [NSObject: AnyObject]?) var persistentStoreCoordinator: NSPersistentStoreCoordinator? { get } } In the end, Dependency Injection as defined by James Shore is all here: define the collaborators with protocols and then pass them in the constructor. This is the best way to do DI. After the construction, the object is fully formed and it has a consistent state. Also, by just looking at the signature of init, the dependencies of this object are clear. Actually, the Constructor Injection is not only the most effective, but it's also the easiest. The only problem is who has to create the object graph? The parent object? The AppDelegate? We'll discuss that point in the Where to bind the dependencies section. Property Injection We have already agreed that Construction Injection is the best way to do DI, so why bother finding other methods? Well, it is not always possible to define the constructor the way we want. A notable example is doing DI with ViewControllers that are defined in storyboards. Given we have a BasketViewController that orchestrates the service and the store, we must pass them as properties: class BasketViewController: UIViewController { var service: BasketService? var store: BasketStore? // ... } This pattern is less elegant than the previous one: The ViewController isn't in the right state until all the properties are set Properties introduce mutability, and immutable classes are simpler and more efficient The properties must be defined as optional, leading to add question marks everywhere They are set by an external object, so they must be writeable and this could potentially permit something else to overwrite the value set at the beginning after a while There is no way to enforce the validity of the setup at compile-time However, something can be done: The properties can be set as implicitly unwrapped optional and then required in viewDidLoad. This is as a static check, but at least they are checked at the first sensible opportunity, which is when the view controller has been loaded. A function setter of all the properties prevents us from partially defining the collaborator list. The class BasketViewController must then be written as: class BasketViewController: UIViewController { private var service: BasketService! private var store: BasketStore! func set(service: BasketService, store: BasketStore) { self.service = service self.store = store } override func viewDidLoad() { super.viewDidLoad() precondition(service != nil, "BasketService required") precondition(store != nil, "BasketStore required") // ... } } The Properties Injection permits us to have overridable properties with a default value. This can be useful in the case of testing. Let's consider a dependency to a wrapper around the time: class CheckoutViewController: UIViewController { var time: Time = DefaultTime() } protocol Time { func now() -> Date } struct DefaultTime: Time { func now() -> Date { return Date() } } In the production code, we don't need to do anything, while in the testing code we can now inject a particular date instead of always return the current time. This would permit us of testing how the software will behave in the future, or in the past. A dependency defined in the same module or framework is Local. When it comes from another module or framework, it's Foreign. A Local dependency can be used as a default value, but a Foreign cannot, otherwise it would introduce a strong dependency between the modules. Method Injection This pattern just passes a collaborator in the method: class BasketClient { func add(product: Product, to store: BasketStore) { store.add(product: product) calculateAppliedDiscount() //... } // ... private func calculateAppliedDiscount() { // ... } } This is useful when the object has several collaborators, but most of them are just temporary and it isn't worth having the relationship set up for the whole life cycle of the object. Ambient Context The final pattern, Ambient Context, is similar to the Singleton. We still have a single instance as a static variable, but the class has multiple subclasses with different behaviors, and each static variable is writeable with a static function: class Analytics { static private(set) var instance: Analytics = NoAnalytics() static func setAnaylics(analitics: Analytics) { self.instance = analitics } func track(event: Event) { fatalError("Implement in a subclass") } } class NoAnalytics: Analytics { override func track(event: Event) {} } class GoogleAnalytics: Analytics { override func track(event: Event) { //... } } class AdobeAnalytics: Analytics { override func track(event: Event) { //... } } struct Event { //... } This pattern should be used only for universal dependencies, representing some cross-cutting concerns, such as analytics, logging, and times and dates. This pattern has some advantages. The dependencies are always accessible and don't need to change the API. It works well for cross-cutting concerns, but it doesn't fit in other cases when the object isn't unique. Also, it makes the dependency implicit and it represents a global mutable state that sometimes can lead to issues that are difficult to debug. DI anti-patterns When we try to implement a new technique, it is quite easy to lose control and implement it in the wrong way. Let's see then the most common anti-patterns in Dependency Injection. Control Freak The first one is pretty easy to spot: we are not using the Injection at all. Instead of being Injected, the dependency is instantiated inside the object that depends on it: class FeaturedProductsController { private let restProductsService: ProductsService init() { self.restProductsService = RestProductsService(configuration: Configuration.loadFromBundleId()) } } In this example, ProductsService could have been injected in the constructor but it is instantiated there instead. Mark Seeman, in his book Dependency Injection in .NET, Chapter 5.1 - DI anti-patterns, calls it Control Freak because it describes a class that will not relinquish its dependencies. The Control Freak is the dominant DI anti-pattern and it happens every time a class directly instantiates its dependencies, instead of relying on the Inversion of Control for that. In the case of the example, even though the rest of the class is programmed against an interface, there is no way of changing the actual implementation of ProductsService and the type of concrete class that it is, it will always be RestProductsService. The only way to change it is to modify the code and compile it again, but with DI it should be possible to change the behavior at runtime. Sometimes, someone tries to fix the Control Freak anti-pattern using the factory pattern, but the reality is that the only way to fix it is to apply the Inversion of Control for the dependency and inject it in the constructor: class FeaturedProductsController { private let productsService: ProductsService init(service: ProductsService) { self.productsService = service } } As already mentioned, Control Freak is the most common DI anti-pattern; pay particular attention so you don't slip into its trap. Bastard Injection Constructor overloads are fairly common in Swift codebases, but these could lead to the Bastard Injection anti-pattern. A common scenario is when we have a constructor that lets us inject a Test Double, but it also has a default parameter in the constructor: class TodosService { let repository: TodosRepository init(repository: TodosRepository = SqlLiteTodosRepository()) { self.repository = repository } } The biggest problem here is when the default implementation is a Foreign dependency, which is a class defined using another module; this creates a strong relationship between the two modules, making it impossible to reuse the class without including the dependent module too. The reason someone is tempted to write a default implementation it is pretty obvious since it is an easy way to instantiate the class just with TodoService() without the need of Composition Root or something similar. However, this nullifies the benefits of DI and it should be avoided removing the default implementation and injecting the dependency. Service Locator The final anti-pattern that we will explore is the most dangerous one: the Service Locator. It's funny because this is often considered a good pattern and is widely used, even in the famous Spring framework. Originally, the Service Locator pattern was defined in Microsoft patterns & practices' Enterprise Library, as Mark Seeman writes in his book Dependency Injection in .NET, Chapter 5.4 - Service Locator, but now he is advocating strongly against it. Service Locator is a common name for a service that we can query for different objects that were previously registered in it. As mentioned, it is a tricky one because it makes everything seem OK, but in fact, it nullifies all the advantage of the Dependency Injection: let locator = ServiceLocator.instance locator.register( SqlLiteTodosRepository(), forType: TodosRepository.self) class TodosService { private let repository: TodosRepository init() { let locator = ServiceLocator.instance self.repository = locator.resolve(TodosRepository.self) } } Here we have a service locator as a singleton, to whom we register the classes we want to resolve. Instead of injecting the class into the constructor, we just query from the service. It looks like the Service Locator has all the advantages of Dependency Injection, it provides testability and extensibility since we can use different implementations without changing the client. It also enables parallel development and separated configuration from the usage. But it has some major disadvantages. With DI, the dependencies are explicit; it's enough to look at the signature of the constructor or the exposed properties to understand what the dependencies for a class are. With a Service Locator, these dependencies are implicit, and the only way to find them is to inspect the implementation, which breaks the encapsulation. Also, all the classes are depending on the Service Locator and this makes the code tightly coupled with it. If we want to reuse a class, other then that class, we also need to add the Service Locator in our project, which could be in a different module and then adding the whole module as dependency where we wanted just to use one class. Service Locator could also give us the impression that we are not using DI at all because all the dependencies are hidden inside the classes. In this article, we covered the different flavors of dependency injection and examines how each can solve a particular set of problems in real-world scenarios. If you found this post useful, do check out the book, Hands-On Design Patterns with Swift. From learning about the most sought-after design patterns to comprehensive coverage of architectural patterns and code testing, this book is all you need to write clean, reusable code in Swift. Implementing Dependency Injection in Google Guice [Tutorial] Implementing Dependency Injection in Spring [Tutorial] Dagger 2.17, a dependency injection framework for Java and Android, is now out!
Read more
  • 0
  • 0
  • 6136

article-image-feature-improvement-identifying-missing-values-using-eda-exploratory-data-analysis-technique
Pravin Dhandre
13 Mar 2018
9 min read
Save for later

Feature Improvement: Identifying missing values using EDA (Exploratory Data Analysis) technique

Pravin Dhandre
13 Mar 2018
9 min read
Today, we will work towards developing a better sense of data through identifying missing values in a dataset using Exploratory Data Analysis (EDA) technique and python packages. Identifying missing values in data Our first method of identifying missing values is to give us a better understanding of how to work with real-world data. Often, data can have missing values due to a variety of reasons, for example with survey data, some observations may not have been recorded. It is important for us to analyze our data, and get a sense of what the missing values are so we can decide how we want to handle missing values for our machine learning. To start, let's dive into a dataset the Pima Indian Diabetes Prediction dataset. This dataset is available on the UCI Machine Learning Repository at: https:/​/​archive.​ics.​uci.​edu/​ml/​datasets/​pima+indians+diabetes From the main website, we can learn a few things about this publicly available dataset. We have nine columns and 768 instances (rows). The dataset is primarily used for predicting the onset of diabetes within five years in females of Pima Indian heritage over the age of 21 given medical details about their bodies. The dataset is meant to correspond with a binary (2-class) classification machine learning problem. Namely, the answer to the question, will this person develop diabetes within five years? The column names are provided as follows (in order): Number of times pregnant Plasma glucose concentration a 2 hours in an oral glucose tolerance test Diastolic blood pressure (mm Hg) Triceps skinfold thickness (mm) 2-Hour serum insulin measurement (mu U/ml) Body mass index (weight in kg/(height in m)2) Diabetes pedigree function Age (years) Class variable (zero or one) The goal of the dataset is to be able to predict the final column of class variable, which predicts if the patient has developed diabetes, using the other eight features as inputs to a machine learning function. There are two very important reasons we will be working with this dataset: We will have to work with missing values All of the features we will be working with will be quantitative The first point makes more sense for now as a reason, because the point of this chapter is to deal with missing values. As far as only choosing to work with quantitative data, this will only be the case for this chapter. We do not have enough tools to deal with missing values in categorical columns. In the next chapter, when we talk about feature construction, we will deal with this procedure. The exploratory data analysis (EDA) To identify our missing values we will begin with an EDA of our dataset. We will be using some useful python packages, pandas and numpy, to store our data and make some simple calculations as well as some popular visualization tools to see what the distribution of our data looks like. Let's begin and dive into some code. First, we will do some imports: # import packages we need for exploratory data analysis (EDA) import pandas as pd # to store tabular data import numpy as np # to do some math import matplotlib.pyplot as plt # a popular data visualization tool import seaborn as sns # another popular data visualization tool %matplotlib inline plt.style.use('fivethirtyeight') # a popular data visualization theme We will import our tabular data through a CSV, as follows: # load in our dataset using pandas pima = pd.read_csv('../data/pima.data') pima.head() The head method allows us to see the first few rows in our dataset. The output is as follows: Something's not right here, there's no column names. The CSV must not have the names for the columns built into the file. No matter, we can use the data source's website to fill this in, as shown in the following code: pima_column_names = ['times_pregnant', 'plasma_glucose_concentration', 'diastolic_blood_pressure', 'triceps_thickness', 'serum_insulin', 'bmi', 'pedigree_function', 'age', 'onset_diabetes'] pima = pd.read_csv('../data/pima.data', names=pima_column_names) pima.head() Now, using the head method again, we can see our columns with the appropriate headers. The output of the preceding code is as follows: Much better, now we can use the column names to do some basic stats, selecting, and visualizations. Let's first get our null accuracy as follows: pima['onset_diabetes'].value_counts(normalize=True) # get null accuracy, 65% did not develop diabetes 0 0.651042 1 0.348958 Name: onset_diabetes, dtype: float64 If our eventual goal is to exploit patterns in our data in order to predict the onset of diabetes, let us try to visualize some of the differences between those that developed diabetes and those that did not. Our hope is that the histogram will reveal some sort of pattern, or obvious difference in values between the classes of prediction: # get a histogram of the plasma_glucose_concentration column for # both classes col = 'plasma_glucose_concentration' plt.hist(pima[pima['onset_diabetes']==0][col], 10, alpha=0.5, label='nondiabetes') plt.hist(pima[pima['onset_diabetes']==1][col], 10, alpha=0.5, label='diabetes') plt.legend(loc='upper right') plt.xlabel(col) plt.ylabel('Frequency') plt.title('Histogram of {}'.format(col)) plt.show() The output of the preceding code is as follows: It seems that this histogram is showing us a pretty big difference between plasma_glucose_concentration between the two prediction classes. Let's show the same histogram style for multiple columns as follows: for col in ['bmi', 'diastolic_blood_pressure', 'plasma_glucose_concentration']: plt.hist(pima[pima['onset_diabetes']==0][col], 10, alpha=0.5, label='non-diabetes') plt.hist(pima[pima['onset_diabetes']==1][col], 10, alpha=0.5, label='diabetes') plt.legend(loc='upper right') plt.xlabel(col) plt.ylabel('Frequency') plt.title('Histogram of {}'.format(col)) plt.show() The output of the preceding code will give us the following three histograms. The first one is show us the distributions of bmi for the two class variables (non-diabetes and diabetes): The next histogram to appear will shows us again contrastingly different distributions between a feature across our two class variables. This time we are looking at diastolic_blood_pressure: The final graph will show plasma_glucose_concentration differences between our two class Variables: We can definitely see some major differences simply by looking at just a few histograms. For example, there seems to be a large jump in plasma_glucose_concentration for those who will eventually develop diabetes. To solidify this, perhaps we can visualize a linear correlation matrix in an attempt to quantify the relationship between these variables. We will use the visualization tool, seaborn, which we imported at the beginning of this chapter for our correlation matrix as follows: # look at the heatmap of the correlation matrix of our dataset sns.heatmap(pima.corr()) # plasma_glucose_concentration definitely seems to be an interesting feature here Following is the correlation matrix of our dataset. This is showing us the correlation amongst the different columns in our Pima dataset. The output is as follows: This correlation matrix is showing a strong correlation between plasma_glucose_concentration and onset_diabetes. Let's take a further look at the numerical correlations for the onset_diabetes column, with the following code: pima.corr()['onset_diabetes'] # numerical correlation matrix # plasma_glucose_concentration definitely seems to be an interesting feature here times_pregnant 0.221898 plasma_glucose_concentration 0.466581 diastolic_blood_pressure 0.065068 triceps_thickness 0.074752 serum_insulin 0.130548 bmi 0.292695 pedigree_function 0.173844 age 0.238356 onset_diabetes 1.000000 Name: onset_diabetes, dtype: float64 We will explore the powers of correlation in a later Chapter 4, Feature Construction, but for now we are using exploratory data analysis (EDA) to hint at the fact that the plasma_glucose_concentration column will be an important factor in our prediction of the onset of diabetes. Moving on to more important matters at hand, let's see if we are missing any values in our dataset by invoking the built-in isnull() method of the pandas DataFrame: pima.isnull().sum() >>>> times_pregnant 0 plasma_glucose_concentration 0 diastolic_blood_pressure 0 triceps_thickness 0 serum_insulin 0 bmi 0 pedigree_function 0 age 0 onset_diabetes 0 dtype: int64 Great! We don't have any missing values. Let's go on to do some more EDA, first using the shape method to see the number of rows and columns we are working with: pima.shape . # (# rows, # cols) (768, 9) Confirming we have 9 columns (including our response variable) and 768 data observations (rows). Now, let's take a peak at the percentage of patients who developed diabetes, using the following code: pima['onset_diabetes'].value_counts(normalize=True) # get null accuracy, 65% did not develop diabetes 0 0.651042 1 0.348958 Name: onset_diabetes, dtype: float64 This shows us that 65% of the patients did not develop diabetes, while about 35% did. We can use a nifty built-in method of a pandas DataFrame called describe to look at some basic descriptive statistics: pima.describe() # get some basic descriptive statistics We get the output as follows: This shows us quite quickly some basic stats such as mean, standard deviation, and some different percentile measurements of our data. But, notice that the minimum value of the BMI column is 0. That is medically impossible; there must be a reason for this to happen. Perhaps the number zero has been encoded as a missing value instead of the None value or a missing cell. Upon closer inspection, we see that the value 0 appears as a minimum value for the following columns: times_pregnant plasma_glucose_concentration diastolic_blood_pressure triceps_thickness serum_insulin bmi onset_diabetes Because zero is a class for onset_diabetes and 0 is actually a viable number for times_pregnant, we may conclude that the number 0 is encoding missing values for: plasma_glucose_concentration diastolic_blood_pressure triceps_thickness serum_insulin bmi So, we actually do having missing values! It was obviously not luck that we happened upon the zeros as missing values, we knew it beforehand. As a data scientist, you must be ever vigilant and make sure that you know as much about the dataset as possible in order to find missing values encoded as other symbols. Be sure to read any and all documentation that comes with open datasets in case they mention any missing values. If no documentation is available, some common values used instead of missing values are: 0 (for numerical values) unknown or Unknown (for categorical variables) ? (for categorical variables) To summarize, we have five columns where the fields are left with missing values and symbols. [box type="note" align="" class="" width=""]You just read an excerpt from a book Feature Engineering Made Easy co-authored by Sinan Ozdemir and Divya Susarla. To learn more about missing values and manipulating features, do check out Feature Engineering Made Easy and develop expert proficiency in Feature Selection, Learning, and Optimization.[/box]    
Read more
  • 0
  • 0
  • 6135
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at $15.99/month. Cancel anytime
article-image-how-to-create-observables-in-rxjs-tutorial
Sugandha Lahoti
10 Apr 2019
7 min read
Save for later

How to create observables in RxJS [Tutorial]

Sugandha Lahoti
10 Apr 2019
7 min read
Reactive programming requires us to change the way that we think about events in an application. Reactive programming requires us to think about events as a stream of values. For example, a mouse click event can be represented as a stream of data. Every click event generates a new value in the data stream. In reactive programming, we can use the stream of data to query and manipulate the values in the stream. Observables are streams of data, and this explains why it is easy to imagine that we can represent an event such as an onClick event using an observable. However, the use cases for observables are much more diverse than that. In this article, we are going to explore how to create an observable given different types. This article is taken from the book Hands-On Functional Programming with TypeScript by Remo H. Jansen. In this book, you will discover the power of functional programming, lazy evaluation, monads, concurrency, and immutability to create succinct and expressive implementations. Creating observables from a value We can create an observable given a value using the of function. In the old versions of RxJS, the function of was a static method of the Observable class, which was available as Observable.of. This should remind us to use the of method of the Applicative type in category theory because observables take some inspiration from category theory. However, in RxJS 6.0, the of method is available as a standalone factory function: import { of } from "rxjs"; const observable = of(1); const subscription = observable.subscribe( (value) => console.log(value), (error: any) => console.log(error), () => console.log("Done!") ); subscription.unsubscribe(); The preceding code snippet declares an observable with one unique value using the of function. The code snippet also showcases how we can subscribe to an observable using the subscribe method. The subscribe method takes three function arguments: Item handler: Invoked once for each item in the sequence. Error handler: Invoked if there is an error in the sequence. This argument is optional. Done handler: Invoked when there are no more items in the sequence. This argument is optional. The following diagram is known as a marble diagram and is used to represent observables in a visual manner. The arrow represents the time and the circles are values. In this case, we have only one value: As we can see, the circle also has a small vertical line in the middle. This line is used to represent the last element in an observable. In this case, the item handler in the subscription will only be invoked once. Creating observables from arrays We can create an observable given an existing array using the from function: import { from } from "rxjs"; const observable = from([10, 20, 30]); const subscription = observable.subscribe( (value) => console.log(value), (error: any) => console.log(error), () => console.log("Done!") ); subscription.unsubscribe(); The preceding code snippet declares an observable with three values using the from function. The code snippet also showcases how we can subscribe once more. The following marble diagram represents the preceding example in a visual manner. The generated observable has three values (10, 20, and 30) and 30 is the last element in the observable: We can alternatively use the interval function to generate an array with a given number of elements: import { interval } from "rxjs"; const observable = interval(10); const subscription = observable.subscribe( (value) => console.log(value), (error: any) => console.log(error), () => console.log("Done!") ); subscription.unsubscribe(); The preceding code snippet declares an observable with ten values using the interval function. The code snippet also showcases how we can subscribe once more. In this case, the item handler in the subscription will be invoked ten times. The following marble diagram represents the preceding example in a visual manner. The generating observable has ten values, and 9 is the last item contained by it:  In this case, the item handler in the subscription will be invoked ten times. Creating observables from events It is also possible to create an observable using an event as the source of the items in the stream. We can do this using the fromEvent function: import { fromEvent } from "rxjs"; const observable = fromEvent(document, "click"); const subscription = observable.subscribe( (value) => console.log(value) ); subscription.unsubscribe(); In this case, the item handler in the subscription will be invoked as many times as the click event takes place. Please note that the preceding example can only be executed in a web browser. To execute the preceding code in a web browser, you will need to use a module bundler, such as Webpack. Creating observables from callbacks It is also possible to create an observable that will iterate the arguments of a callback using the bindCallback function: import { bindCallback } from "rxjs"; import fetch from "node-fetch"; function getJSON(url: string, cb: (response: unknown|null) => void) { fetch(url) .then(response => response.json()) .then(json => cb(json)) .catch(_ => cb(null)); } const uri = "https://jsonplaceholder.typicode.com/todos/1"; const observableFactory = bindCallback(getJSON); const observable = observableFactory(uri); const subscription = observable.subscribe( (value) => console.log(value) ); subscription.unsubscribe(); The preceding example uses the node-fetch module because the fetch function is not available in Node.js. You can install the node-fetch module using the following npm command: npm install node-fetch @types/node-fetch The getJSON function takes a URL and a callback as its arguments. When we pass it to the bindCallback function, a new function is returned. The new function takes a URL as its only argument and returns an observable instead of taking a callback. In Node.js, callbacks follow a well-defined pattern. The Node.js callbacks take two arguments, error and result, and don't throw exceptions. We must use the error argument to check whether something went wrong instead of a try/catch statement. RxJS also defines a function named bindNodeCallback that allows us to work with the callbacks: import { bindNodeCallback } from "rxjs"; import * as fs from "fs"; const observableFactory = bindNodeCallback(fs.readFile); const observable = observableFactory("./roadNames.txt"); const subscription = observable.subscribe( (value) => console.log(value.toString()) ); subscription.unsubscribe(); The helpers, bindCallback and bindNodeCallback, have very similar behavior, but the second has been specially designed to work with Node.js callbacks. Creating observables from promises Another potential source of items for an observable sequence is a Promise. RxJS also allows us to handle this use case with the from function. We must pass a Promise instance to the from function. In the following example, we use the fetch function to send an HTTP request. The fetch function returns a promise that is passed to the from function: import { bindCallback } from "rxjs"; import fetch from "node-fetch"; const uri = "https://jsonplaceholder.typicode.com/todos/1"; const observable = from(fetch(uri)).pipe(map(x => x.json())); const subscription = observable.subscribe( (value) => console.log(value.toString()) ); subscription.unsubscribe(); The generated observable will contain the result of the promise as its only item. Cold and hot observables The official RxJS documentation explores the differences between cold and hot observables as follows: "Cold observables start running upon subscription, that is, the observable sequence only starts pushing values to the observers when Subscribe is called. Values are also not shared among subscribers. This is different from hot observables, such as mouse move events or stock tickers, which are already producing values even before a subscription is active. When an observer subscribes to a hot observable sequence, it will get all values in the stream that are emitted after it subscribes. The hot observable sequence is shared among all subscribers, and each subscriber is pushed to the next value in the sequence." It is important to understand these differences if we want to have control over the execution flow of our components. The key point to remember is that cold observables are lazily evaluated. In this article, we learned what observables are and how we can create them and work with them. To know more about working with observables, and other aspects of functional programming, read our book Hands-On Functional Programming with TypeScript. What makes functional programming a viable choice for artificial intelligence projects? Why functional programming in Python matters: Interview with best selling author, Steven Lott Introducing Coconut for making functional programming in Python simpler
Read more
  • 0
  • 0
  • 6127

article-image-using-protocols-and-protocol-extensions
Packt
13 Oct 2015
22 min read
Save for later

Using Protocols and Protocol Extensions

Packt
13 Oct 2015
22 min read
In this article by Jon Hoffman, the author of Mastering Swift 2, we'll see how protocols are used as a type, how we can implement polymorphism in Swift using protocols, how to use protocol extensions, and why we would want to use protocol extensions. (For more resources related to this topic, see here.) While watching the presentations from WWDC 2015 about protocol extensions and protocol-oriented programming, I will admit that I was very skeptical. I have worked with object-oriented programming for so long that I was unsure if this new programming paradigm would solve all of the problems that Apple was claiming it would. Since I am not one to let my skepticism get in the way of trying something new, I set up a new project that mirrored the one I was currently working on, but wrote the code using Apple's recommendations for protocol-oriented programming and used protocol extensions extensively in the code. I can honestly say that I was amazed with how much cleaner the new project was compared to the original one. I believe that protocol extensions is going to be one of those defining features that set one programming language apart from the rest. I also believe that many major languages will soon have similar features. Protocol extensions are the backbone for Apple's new protocol-oriented programming paradigm and is arguably one of the most important additions to the Swift programming language. With protocol extensions, we are able to provide method and property implementations to any type that conforms to a protocol. To really understand how useful protocols and protocol extensions are, let's get a better understanding of protocols. While classes, structs, and enums can all conform to protocols in Swift, for this article, we will be focusing on classes and structs. Enums are used when we need to represent a finite number of cases and while there are valid use cases where we would have an enum conform to a protocol, they are very rare in my experience. Just remember that anywhere that we refer to a class or struct, we can also use an enum. Let's begin exploring protocols by seeing how they are full-fledged types in Swift. Protocol as a type Even though no functionality is implemented in a protocol, they are still considered a full-fledged type in the Swift programming language and can be used like any other type. What this means is we can use protocols as a parameter type or a return type in a function. We can also use them as the type for variables, constants, and collections. Let's take a look at some examples. For these few examples, we will use the PersonProtocol protocol: protocol PersonProtocol { var firstName: String {get set} var lastName: String {get set} var birthDate: NSDate {get set} var profession: String {get} init (firstName: String, lastName: String, birthDate: NSDate) } In this first example, we will see how we would use protocols as a parameter type or return type in functions, methods, or initializers: func updatePerson(person: PersonProtocol) -> PersonProtocol { // Code to update person goes here return person } In this example, the updatePerson() function accepts one parameter of the PersonProtocol protocol type and then returns a value of the PersonProtocol protocol type. Now let's see how we can use protocols as a type for constants, variables, or properties: var myPerson: PersonProtocol In this example, we create a variable of the PersonProtocol protocol type that is named myPerson. We can also use protocols as the item type to store in collection such as arrays, dictionaries, or sets: var people: [PersonProtocol] = [] In this final example, we create an array of PersonProtocol protocol types. As we can see from these three examples, even though the PersonProtocol protocol does not implement any functionality, we can still use protocols when we need to specify a type. We cannot, however, create an instance of a protocol. This is because no functionality is implemented in a protocol. As an example, if we tried to create an instance of the PersonProtocol protocol, we would be receiving the error: protocol type 'PersonProtocol' cannot be instantiated error, as shown in the following example: var test = PersonProtocol(firstName: "Jon", lastName: "Hoffman", birthDate: bDateProgrammer) We can use the instance of any class or struct that conforms to our protocol anywhere that the protocol type is required. As an example, if we defined a variable to be of the PersonProtocol protocol type, we could then populate that variable with any class or struct that conforms to the PersonProtocol protocol. For this example, let's assume that we have two types named SwiftProgrammer and FootballPlayer, which conform to the PersonProtocol protocol: var myPerson: PersonProtocol myPerson = SwiftProgrammer(firstName: "Jon", lastName: "Hoffman", birthDate: bDateProgrammer) print("(myPerson.firstName) (myPerson.lastName)") myPerson = FootballPlayer(firstName: "Dan", lastName: "Marino", birthDate: bDatePlayer) print("(myPerson.firstName) (myPerson.lastName)") In this example, we start off by creating the myPerson variable of the PersonProtocol protocol type. We then set the variable with an instance of the SwiftProgrammer type and print out the first and last names. Next, we set the myPerson variable to an instance of the FootballPlayer type and print out the first and last names again. One thing to note is that Swift does not care if the instance is a class or struct. It only matters that the type conforms to the PersonProtocol protocol type. Therefore, if our SwiftProgrammer type was a struct and the FootballPlayer type was a class, our previous example would be perfectly valid. As we saw earlier, we can use our PersonProtocol protocol as the type for an array. This means that we can populate the array with instances of any type that conforms to the PersonProtocol protocol. Once again, it does not matter if the type is a class or a struct as long as it conforms to the PersonProtocol protocol. Here is an example of this: var programmer = SwiftProgrammer(firstName: "Jon", lastName: "Hoffman", birthDate: bDateProgrammer) var player = FootballPlayer(firstName: "Dan", lastName: "Marino", birthDate: bDatePlayer) var people: [PersonProtocol] = [] people.append(programmer) people.append(player) In this example, we create an instance of the SwiftProgrammer type and an instance of the FootballPlayer type. We then add both instances to the people array. Polymorphism with protocols What we were seeing in the previous examples is a form of Polymorphism. The word polymorphism comes from the Greek roots Poly, meaning many and morphe, meaning form. In programming languages, polymorphism is a single interface to multiple types (many forms). In the previous example, the single interface was the PersonProtocol protocol and the multiple types were any type that conforms to that protocol. Polymorphism gives us the ability to interact with multiple types in a uniform manner. To illustrate this, we can extend our previous example where we created an array of the PersonProtocol types and loop through the array. We can then access each item in the array using the properties and methods define in the PersonProtocol protocol, regardless of the actual type. Let's see an example of this: for person in people { print("(person.firstName) (person.lastName): (person.profession)") } If we ran this example, the output would look similar to this: Jon Hoffman: Swift Programmer Dan Marino: Football Player We have mentioned a few times in this article that when we define the type of a variable, constant, collection type, and so on to be a protocol type, we can then use the instance of any type that conforms to that protocol. This is a very important concept to understand and it is what makes protocols and protocol extensions so powerful. When we use a protocol to access instances, as shown in the previous example, we are limited to using only properties and methods that are defined in the protocol. If we want to use properties or methods that are specific to the individual types, we would need to cast the instance to that type. Type casting with protocols Type casting is a way to check the type of the instance and/or to treat the instance as a specified type. In Swift, we use the is keyword to check if an instance is a specific type and the as keyword to treat the instance as a specific type. To start with, let's see how we would check the instance type using the is keyword. The following example shows how would we do this: for person in people { if person is SwiftProgrammer { print("(person.firstName) is a Swift Programmer") } } In this example, we use the if conditional statement to check whether each element in the people array is an instance of the SwiftProgrammer type and if so, we print that the person is a Swift programmer to the console. While this is a good method to check whether we have an instance of a specific class or struct, it is not very efficient if we wanted to check for multiple types. It is a lot more efficient to use the switch statement, as shown in the next example, if we want to check for multiple types: for person in people { switch (person) { case is SwiftProgrammer: print("(person.firstName) is a Swift Programmer") case is FootballPlayer: print("(person.firstName) is a Football Player") default: print("(person.firstName) is an unknown type") } } In the previous example, we showed how to use the switch statement to check the instance type for each element of the array. To do this check, we use the is keyword in each of the case statements in an attempt to match the instance type. We can also use the where statement with the is keyword to filter the array, as shown in the following example: for person in people where person is SwiftProgrammer { print("(person.firstName) is a Swift Programmer") } Now let's look at how we can cast an instance of a class or struct to a specific type. To do this, we can use the as keyword. Since the cast can fail if the instance is not of the specified type, the as keyword comes in two forms: as? and as!. With the as? form, if the casting fails, it returns a nil, and with the as! form, if the casting fails, we get a runtime error; therefore, it is recommended to use the as? form unless we are absolutely sure of the instance type or we perform a check of the instance type prior to doing the cast. Let's look at how we would use the as? keyword to cast an instance of a class or struct to a specified type: for person in people { if let p = person as? SwiftProgrammer { print("(person.firstName) is a Swift Programmer") } } Since the as? keyword returns an optional, we can use optional binding to perform the cast, as shown in this example. If we are sure of the instance type, we can use the as! keyword. The following example shows how to use the as! keyword when we filter the results of the array to only return instances of the SwiftProgrammer type: for person in people where person is SwiftProgrammer { let p = person as! SwiftProgrammer } Now that we have covered the basics of protocols, that is, how polymorphism works and type casting, let's dive into one of the most exciting new features of Swift protocol extensions. Protocol extensions Protocol extensions allow us to extend a protocol to provide method and property implementations to conforming types. It also allows us to provide common implementations to all the confirming types eliminating the need to provide an implementation in each individual type or the need to create a class hierarchy. While protocol extensions may not seem too exciting, once you see how powerful they really are, they will transform the way you think about and write code. Let's begin by looking at how we would use protocol extension with a very simplistic example. We will start off by defining a protocol called DogProtocol as follows: protocol DogProtocol { var name: String {get set} var color: String {get set} } With this protocol, we are saying that any type that conforms to the DogProtocol protocol, must have the two properties of the String type, namely, name and color. Now let's define the three types that conform to this protocol. We will name these types JackRussel, WhiteLab, and Mutt as follows: struct JackRussel: DogProtocol { var name: String var color: String } class WhiteLab: DogProtocol { var name: String var color: String init(name: String, color: String) { self.name = name self.color = color } } struct Mutt: DogProtocol { var name: String var color: String } We purposely created the JackRussel and Mutt types as structs and the WhiteLab type as a class to show the differences between how the two types are set up and to illustrate how they are treated in the same way when it comes to protocols and protocol extensions. The biggest difference that we can see in this example is the struct types provide a default initiator, but in the class, we must provide the initiator to populate the properties. Now let's say that we want to provide a method named speak to each type that conforms to the DogProtocol protocol. Prior to protocol extensions, we would start off by adding the method definition to the protocol, as shown in the following code: protocol DogProtocol { var name: String {get set} var color: String {get set} func speak() -> String } Once the method is defined in the protocol, we would then need to provide an implementation of the method in every type that conform to the protocol. Depending on the number of types that conformed to this protocol, this could take a bit of time to implement. The following code sample shows how we might implement this method: struct JackRussel: DogProtocol { var name: String var color: String func speak() -> String { return "Woof Woof" } } class WhiteLab: DogProtocol { var name: String var color: String init(name: String, color: String) { self.name = name self.color = color } func speak() -> String { return "Woof Woof" } } struct Mutt: DogProtocol { var name: String var color: String func speak() -> String { return "Woof Woof" } } While this method works, it is not very efficient because anytime we update the protocol, we would need to update all the types that conform to it and we may be duplicating a lot of code, as shown in this example. Another concern is, if we need to change the default behavior of the speak() method, we would have to go in each implementation and change the speak() method. This is where protocol extensions come in. With protocol extensions, we could take the speak() method definition out of the protocol itself and define it with the default behavior, in protocol extension. The following code shows how we would define the protocol and the protocol extension: protocol DogProtocol { var name: String {get set} var color: String {get set} } extension DogProtocol { func speak() -> String { return "Woof Woof" } } We begin by defining DogProtocol with the original two properties. We then create a protocol extension that extends DogProtocol and contains the default implementation of the speak() method. With this code, there is no need to provide an implementation of the speak() method in each of the types that conform to DogProtocol because they automatically receive the implementation as part of the protocol. Let's see how this works by setting our three types that conform to DogProtocol back to their original implementations and they should receive the speak() method from the protocol extension: struct JackRussel: DogProtocol { var name: String var color: String } class WhiteLab: DogProtocol { var name: String var color: String init(name: String, color: String) { self.name = name self.color = color } } struct Mutt: DogProtocol { var name: String var color: String } We can now use each of the types as shown in the following code: let dash = JackRussel(name: "Dash", color: "Brown and White") let lily = WhiteLab(name: "Lily", color: "White") let buddy = Mutt(name: "Buddy", color: "Brown") let dSpeak = dash.speak() // returns "woof woof" let lSpeak = lily.speak() // returns "woof woof" let bSpeak = buddy.speak() // returns "woof woof" As we can see in this example, by adding the speak() method to the DogProtocol protocol extension, we are automatically adding that method to all the types that conform to DogProtocol. The speak() method in the DogProtocol protocol extension can be considered a default implementation of the speak() method because we are able to override it in the type implementations. As an example, we could override the speak() method in the Mutt struct, as shown in the following code: struct Mutt: DogProtocol { var name: String var color: String func speak() -> String { return "I am hungry" } } When we call the speak() method for an instance of the Mutt type, it will return the string, "I am hungry". Now that we have seen how we would use protocols and protocol extensions, let's look at a more real-world example. In numerous apps, across multiple platforms (iOS, Android, and Windows), I have had the requirement to validate user input as it is entered. This validation can be done very easily with regular expressions; however, we do not want various regular expressions littered through out our code. It is very easy to solve this problem by creating different classes or structs that contains the validation code; however, we would have to organize these classes to make them easy to use and maintain. Prior to protocol extensions in Swift, I would use protocols to define the validation requirements and then create a struct that would conform to the protocol for each validation that I needed. Let's take a look at this preprotocol extension method. A regular expression is a sequence of characters that define a particular pattern. This pattern can then be used to search a string to see whether the string matches the pattern or contains a match of the pattern. Most major programming languages contain a regular expression parser, and if you are not familiar with regular expressions, it may be worth to learn more about them. The following code shows the TextValidationProtocol protocol that defines the requirements for any type that we want to use for text validation: protocol TextValidationProtocol { var regExMatchingString: String {get} var regExFindMatchString: String {get} var validationMessage: String {get} func validateString(str: String) -> Bool func getMatchingString(str: String) -> String? } In this protocol, we define three properties and two methods that any type that conforms to TextValidationProtocol must implement. The three properties are: regExMatchingString: This is a regular expression string used to verify that the input string contains only valid characters. regExFindMatchString: This is a regular expression string used to retrieve a new string from the input string that contains only valid characters. This regular expression is generally used when we need to validate the input real time, as the user enters information, because it will find the longest matching prefix of the input string. validationMessage: This is the error message to display if the input string contains non-valid characters. The two methods for this protocol are as follows: validateString: This method will return true if the input string contains only valid characters. The regExMatchingString property will be used in this method to perform the match. getMatchingString: This method will return a new string that contains only valid characters. This method is generally used when we need to validate the input real time as the user enters information because it will find the longest matching prefix of the input string. We will use the regExFindMatchString property in this method to retrieve the new string. Now let's see how we would create a struct that conforms to this protocol. The following struct would be used to verify that the input string contains only alpha characters: struct AlphaValidation1: TextValidationProtocol { static let sharedInstance = AlphaValidation1() private init(){} let regExFindMatchString = "^[a-zA-Z]{0,10}" let validationMessage = "Can only contain Alpha characters" var regExMatchingString: String { get { return regExFindMatchString + "$" } } func validateString(str: String) -> Bool { if let _ = str.rangeOfString(regExMatchingString, options: .RegularExpressionSearch) { return true } else { return false } } func getMatchingString(str: String) -> String? { if let newMatch = str.rangeOfString(regExFindMatchString, options: .RegularExpressionSearch) { return str.substringWithRange(newMatch) } else { return nil } } } In this implementation, the regExFindMatchString and validationMessage properties are stored properties, and the regExMatchingString property is a computed property. We also implement the validateString() and getMatchingString() methods within the struct. Normally, we would have several different types that conform to TextValidationProtocol where each one would validate a different type of input. As we can see from the AlphaValidation1 struct, there is a bit of code involved with each validation type. A lot of the code would also be duplicated in each type. The code for both methods (validateString() and getMatchingString()) and the regExMatchingString property would be duplicated in every validation class. This is not ideal, but if we wanted to avoid creating a class hierarchy with a super class that contains the duplicate code (I personally prefer using value types over classes), we would have no other choice. Now let's see how we would implement this using protocol extensions. With protocol extensions we need to think about the code a little differently. The big difference is, we do not need, nor want to define everything in the protocol. With standard protocols or when we use class hierarchy, all the methods and properties that you would want to access using the generic superclass or protocol would have to be defined within the superclass or protocol. With protocol extensions, it is preferred for us not to define a property or method in the protocol if we are going to be defining it within the protocol extension. Therefore, when we rewrite our text validation types with protocol extensions, TextValidationProtocol would be greatly simplified to look similar to this: protocol TextValidationProtocol { var regExFindMatchString: String {get} var validationMessage: String {get} } In original TextValidationProtocol, we defined three properties and two methods. As we can see in this new protocol, we are only defining two properties. Now that we have our TextValidationProtocol defined, let's create the protocol extension for it: extension TextValidationProtocol { var regExMatchingString: String { get { return regExFindMatchString + "$" } } func validateString(str: String) -> Bool { if let _ = str.rangeOfString(regExMatchingString, options: .RegularExpressionSearch) { return true } else { return false } } func getMatchingString(str: String) -> String? { if let newMatch = str.rangeOfString(regExFindMatchString, options: .RegularExpressionSearch) { return str.substringWithRange(newMatch) } else { return nil } } } In the TextValidationProtocol protocol extension, we define the two methods and the third property that were defined in original TextValidationProtocol, but were not defined in the new one. Now that we have created our protocol and protocol extension, we are able to define our text validation types. In the following code, we define three structs that we will use to validate text when a users types it in: struct AlphaValidation: TextValidationProtocol { static let sharedInstance = AlphaValidation() private init(){} let regExFindMatchString = "^[a-zA-Z]{0,10}" let validationMessage = "Can only contain Alpha characters" } struct AlphaNumericValidation: TextValidationProtocol { static let sharedInstance = AlphaNumericValidation() private init(){} let regExFindMatchString = "^[a-zA-Z0-9]{0,15}" let validationMessage = "Can only contain Alpha Numeric characters" } struct DisplayNameValidation: TextValidationProtocol { static let sharedInstance = DisplayNameValidation() private init(){} let regExFindMatchString = "^[\s?[a-zA-Z0-9\-_\s]]{0,15}" let validationMessage = "Display Name can contain only contain Alphanumeric Characters" } In each one of the text validation structs, we create a static constant and a private initiator so that we can use the struct as a singleton. After we define the singleton pattern, all we do in each type is set the values for the regExFindMatchString and validationMessage properties. Now we have not duplicated the code virtually because even if we could, we would not want to define the singleton code in the protocol extension because we would not want to force that pattern on all the conforming types. To use the text validation classes, we would want to create a dictionary object that would map the UITextField objects to the validation class to use it like this: var validators = [UITextField: TextValidationProtocol]() We could then populate the validators dictionary as shown here: validators[alphaTextField] = AlphaValidation.sharedInstance validators[alphaNumericTextField] = AlphaNumericValidation.sharedInstance validators[displayNameTextField] = DisplayNameValidation.sharedInstance We can now set the EditingChanged event of the text fields to a single method named keyPressed(). To set the edition changed event for each field, we would add the following code to the viewDidLoad() method of our view controller: alphaTextField.addTarget(self, action:Selector("keyPressed:"), forControlEvents: UIControlEvents.EditingChanged) alphaNumericTextField.addTarget(self, action: Selector("keyPressed:"), forControlEvents: UIControlEvents.EditingChanged) displayNameTextField.addTarget(self, action: Selector("keyPressed:"), forControlEvents: UIControlEvents.EditingChanged) Now lets create the keyPressed() method that each text field calls when a user types a character into the field: @IBAction func keyPressed(textField: UITextField) { if let validator = validators[textField] where !validator.validateString(textField.text!) { textField.text = validator.getMatchingString(textField.text!) messageLabel?.text = validator.validationMessage } } In this method, we use the if let validator = validators[textField] statement to retrieve the validator for the particular text field and then we use the where !validator.validateString(textField.text!) statement to validate the string that the user has entered. If the string fails validation, we use the getMatchingString() method to update the text in the text field by removing all the characters from the input string, starting with the first invalid character and then displaying the error message from the text validation class. If the string passes validation, the text in the text field is left unchanged. Summary In this article, we saw that protocols are treated as full-fledged types by Swift. We also saw how polymorphism can be implemented in Swift with protocols. We concluded this article with an in-depth look at protocol extensions and saw how we would use them in Swift. Protocols and protocol extensions are the backbone of Apple's new protocol-oriented programming paradigm. This new model for programming has the potential to change the way we write and think about code. While we did not specifically cover protocol-oriented programming in this article, understanding the topics in this article gives us the solid understanding of protocols and protocol extensions needed to learn about this new programming model. Resources for Article: Further resources on this subject: Using OpenStack Swift [Article] Dragging a CCNode in Cocos2D-Swift [Article] Installing OpenStack Swift [Article]
Read more
  • 0
  • 0
  • 6125

article-image-websphere-mq-sample-programs
Packt
25 Aug 2010
5 min read
Save for later

WebSphere MQ Sample Programs

Packt
25 Aug 2010
5 min read
(For more resources on IBM, see here.) WebSphere MQ sample programs—server There is a whole set of sample programs available to the WebSphere MQ administrator. We are interested in only a couple of them: the program to put a message onto a queue and a program to retrieve a message from a queue. The names of these sample programs depend on whether we are running WebSphere MQ as a server or as a client. There is a server sample program called amqsput, which puts messages onto a queue using the MQPUT call. This sample program comes as part on the WebSphere MQ samples installation package, not as part of the base package. The maximum length of message that we can put onto a queue using the amqsput sample program is 100 bytes. There is a corresponding server sample program to retrieve messages from a queue called amqsget. Use the sample programs only when testing that the queues are set up correctly—do NOT use the programs once Q Capture and Q Apply have been started. This is because Q replication uses dense numbering between Q Capture and Q Apply, and if we insert or retrieve a message, then the dense numbering will not be maintained and Q Apply will stop. The usual response to this is to cold start Q Capture! To put a message onto a Queue (amqsput) The amqsput utility can be invoked from the command line or from within a batch program. If we are invoking the utility from the command line, the format of the command is: $ amqsput <Queue> <QM name> < <message> We would issue this command from the system on which the Queue Manager sits. We have to specify the queue (<Queue>) that we want to put the message on, and the Queue Manager (<QM name>) which controls this queue. We then pipe (<) the message (<message>) into this. An example of the command is: $ amqsput CAPA.TO.APPB.SENDQ.REMOTE QMA < hello We put these commands into an appropriately named batch fle (say SYSA_QMA_TESTP_UNI_AB.BAT), which would contain the following: Batch file—Windows example: call "C:Program FilesIBMWebSphere MQbinamqsput" CAPA.TO.APPB.SENDQ.REMOTE QMA < QMA_TEST1.TXT Batch file—UNIX example: "/opt/mqm/samp/bin/amqsput" CAPA.TO.APPB.SENDQ.REMOTE QMA < QMA_TEST1.TXT Where the QMA_TEST1.TXT fle contains the message we want to send. Once we have put a message onto the Send Queue, we need to be able to retrieve it. To retrieve a message from a Queue(amqsget) The amqsget utility can be invoked from the command line or from within a batch program. The utility takes 15 seconds to run. We need to specify the Receive Queue that we want to read from and the Queue Manager that the queue belongs to: $ amqsget <Queue> <QM name> As example of the command is shown here: $ amqsget CAPA.TO.APPB.RECVQ QMB If we have correctly set up all the queues, and the Listeners and Channels are running, then when we issue the preceding command, we should see the message we put onto the Send Queue. We can put the amqsget command into a batch fle, as shown next: Batch file—Windows example: @ECHO This takes 15 seconds to runcall "C:Program FilesIBMWebSphere MQbinamqsget" CAPA.TO.APPB.RECVQ QMB@ECHO You should see: test1 Batch file—UNIX example: echo This takes 15 seconds to run"/opt/mqm/samp/bin/amqsget" CAPA.TO.APPB.RECVQ QMBecho You should see: test1 Using these examples and putting messages onto the queues in a unidirectional scenario, then the "get" message batch fle for QMA (SYSA_QMA_TESTG_UNI_BA.BAT) contains: @ECHO This takes 15 seconds to runcall "C:Program FilesIBMWebSphere MQbinamqsget" CAPA.ADMINQ QMA@ECHO You should see: test2 From CLP-A, run the fle as: $ SYSA_QMA_TESTG_UNI_BA.BAT The "get" message batch file for QMB (SYSB_QMB_TESTG_UNI_AB.BAT) contains: @ECHO This takes 15 seconds to runcall "C:Program FilesIBMWebSphere MQbinamqsget" CAPA.TO.APPB.RECVQ QMB@ECHO You should see: test1 From CLP-B, run the file as: $ SYSB_QMB_TESTG_UNI_AB.BAT To browse a message on a Queue It is useful to be able to browse a queue, especially when setting up Event Publishing. There are three ways to browse the messages on a Local Queue. We can use the rfhutil utility, or the amqsbcg sample program, both of which are WebSphere MQ entities, or we can use the asnqmfmt Q replication command. Using the WebSphere MQ rfhutil utility: The rfhutil utility is part of the WebSphere MQ support pack available to download from the web—to find the current download website, simply type rfhutil into an internet search engine. The installation is very simple—unzip the file and run the rfhutil.exe file. Using the WebSphere MQ amqsbcg sample program: The amqsbcg sample program displays messages including message descriptors. $ amqsbcg CAPA.TO.APPB.RECVQ QMB
Read more
  • 0
  • 0
  • 6120

article-image-null-10
Packt
27 Nov 2011
2 min read
Save for later

PacktLib now Offers a Joomla! Library

Packt
27 Nov 2011
2 min read
Packt has today announced a new subscription on PacktLib for Joomla! developers. Housing 26 books, this library will enable Joomla! developers to get up and running quickly, as well as extend their skills and knowledge to become serious professionals. Recently announced as the winner of the best 2011 Open Source CMS, a resurgent Joomla!, now with a six month development cycle, has proved itself to be one of the leading open source content management systems on the market. The Joomla! library consists of information relating to the development of building media rich, dynamic and interactive Joomla! websites. The extensive variety of topics within this open source technology includes eCommerce, search engine optimisation, content administration, multimedia, themes and templates, creating a social network with JomSocial, enhancing a Joomla! site with JavaScript and jQuery, interacting with users by utilising ChronoForms, web security and much more. This source of Joomla! books will enable readers to build powerful websites and online applications by way of add-ons, extensions and plugins in addition to understanding the framework in which this technology is based on. This resource is ideal for beginners who want to learn more about building a functional and well-designed website. Many books in this library provide step-by-step instructions as well as examples to guide readers in setting up a functional and visually appealing website. In addition to this, those who have experience with using Joomla! can find more advanced features to install onto their website making this library a comprehensive resource on Joomla! for readers with different skill levels. Packt was the first publisher to bring an English book on Joomla! to market and has long been a supporter of Joomla! through the Open Source Royalty scheme. With 26 books on this technology, subscribers of the Joomla! library will have full access to an archive of up-to-date information on the content management system. Moreover, future books added to the library will also be made available to subscribers as long as they have a subscription. This resource will give site administrators the tools to create and develop successful websites as well as the ability to search for specific information and find answers to problems on the PacktLib platform. For more information, please visit http://packtlib.packtpub.com/
Read more
  • 0
  • 0
  • 6116
article-image-how-to-stream-and-store-tweets-in-apache-kafka
Fatema Patrawala
22 Dec 2017
8 min read
Save for later

How to stream and store tweets in Apache Kafka

Fatema Patrawala
22 Dec 2017
8 min read
[box type="note" align="" class="" width=""]This article is an excerpt from a book authored by Ankit Jain titled Mastering Apache Storm. This book explores various real-time processing functionalities offered by Apache Storm such as parallelism, data partitioning, and more.[/box] Today, we are going to cover how to stream tweets from Twitter using the twitter streaming API. We are also going to explore how we can store fetched tweets in Kafka for later processing through Storm. Setting up a single node Kafka cluster Following are the steps to set up a single node Kafka cluster:   Download the Kafka 0.9.x binary distribution named kafka_2.10-0.9.0.1.tar.gz from http://apache.claz.org/kafka/0.9.0. or 1/kafka_2.10-0.9.0.1.tgz. Extract the archive to wherever you want to install Kafka with the following command: tar -xvzf kafka_2.10-0.9.0.1.tgz cd kafka_2.10-0.9.0.1   Change the following properties in the $KAFKA_HOME/config/server.properties file: log.dirs=/var/kafka- logszookeeper.connect=zoo1:2181,zoo2:2181,zoo3:2181 Here, zoo1, zoo2, and zoo3 represent the hostnames of the ZooKeeper nodes. The following are the definitions of the important properties in the server.properties file: broker.id: This is a unique integer ID for each of the brokers in a Kafka cluster. port: This is the port number for a Kafka broker. Its default value is 9092. If you want to run multiple brokers on a single machine, give a unique port to each broker. host.name: The hostname to which the broker should bind and advertise itself. log.dirs: The name of this property is a bit unfortunate as it represents not the log directory for Kafka, but the directory where Kafka stores the actual data sent to it. This can take a single directory or a comma-separated list of directories to store data. Kafka throughput can be increased by attaching multiple physical disks to the broker node and specifying multiple data directories, each lying on a different disk. It is not much use specifying multiple directories on the same physical disk, as all the I/O will still be happening on the same disk. num.partitions: This represents the default number of partitions for newly created topics. This property can be overridden when creating new topics. A greater number of partitions results in greater parallelism at the cost of a larger number of files. log.retention.hours: Kafka does not delete messages immediately after consumers consume them. It retains them for the number of hours defined by this property so that in the event of any issues the consumers can replay the messages from Kafka. The default value is 168 hours, which is 1 week. zookeeper.connect: This is the comma-separated list of ZooKeeper nodes in hostname:port form.    Start the Kafka server by running the following command: > ./bin/kafka-server-start.sh config/server.properties [2017-04-23 17:44:36,667] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) [2017-04-23 17:44:36,668] INFO Kafka version : 0.9.0.1 (org.apache.kafka.common.utils.AppInfoParser) [2017-04-23 17:44:36,668] INFO Kafka commitId : a7a17cdec9eaa6c5 (org.apache.kafka.common.utils.AppInfoParser) [2017-04-23 17:44:36,670] INFO [Kafka Server 0], started (kafka.server.KafkaServer) If you get something similar to the preceding three lines on your console, then your Kafka broker is up-and-running and we can proceed to test it. Now we will verify that the Kafka broker is set up correctly by sending and receiving some test messages. First, let's create a verification topic for testing by executing the following command: > bin/kafka-topics.sh --zookeeper zoo1:2181 --replication-factor 1 --partition 1 --topic verification-topic --create Created topic "verification-topic".    Now let's verify if the topic creation was successful by listing all the topics: > bin/kafka-topics.sh --zookeeper zoo1:2181 --list verification-topic    The topic is created; let's produce some sample messages for the Kafka cluster. Kafka comes with a command-line producer that we can use to produce messages: > bin/kafka-console-producer.sh --broker-list localhost:9092 -- topic verification-topic    Write the following messages on your console: Message 1 Test Message 2 Message 3 Let's consume these messages by starting a new console consumer on a new console window: > bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic verification-topic --from-beginning Message 1 Test Message 2 Message 3 Now, if we enter any message on the producer console, it will automatically be consumed by this consumer and displayed on the command line. Collecting Tweets We are assuming you already have a twitter account, and that the consumer key and access token are generated for your application. You can refer to: https://bdthemes.com/support/knowledge-base/generate-api-key-consumer-token-acc ess-key-twitter-oauth/ to generate a consumer key and access token. Take the following steps: Create a new maven project with groupId, com.storm advance and artifactId, kafka_producer_twitter. Add the following dependencies to the pom.xml file. We are adding the Kafka and Twitter streaming Maven dependencies to pom.xml to support the Kafka Producer and the streaming tweets from Twitter: <dependencies> <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka_2.10</artifactId> <version>0.9.0.1</version> <exclusions> <exclusion> <groupId>com.sun.jdmk</groupId> <artifactId>jmxtools</artifactId> </exclusion> <exclusion> <groupId>com.sun.jmx</groupId> <artifactId>jmxri</artifactId> </exclusion> </exclusions> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-slf4j-impl</artifactId> <version>2.0-beta9</version> </dependency> <dependency> <groupId>org.apache.logging.log4j</groupId> <artifactId>log4j-1.2-api</artifactId> <version>2.0-beta9</version> </dependency> <!-- https://mvnrepository.com/artifact/org.twitter4j/twitter4j-stream --> <dependency> <groupId>org.twitter4j</groupId> <artifactId>twitter4j-stream</artifactId> <version>4.0.6</version> </dependency> </dependencies> 3. Now, we need to create a class, TwitterData, that contains the code to consume/stream data from Twitter and publish it to the Kafka cluster. We are assuming you already have a running Kafka cluster and topic, twitterData, created in the Kafka cluster. for information on the installation of the Kafka cluster and the creation of a Kafka please refer to . The class contains an instance of the twitter4j.conf.ConfigurationBuilder class; we need to set the access token and consumer keys in configuration, as mentioned in the source code.4. The twitter4j.StatusListener class returns the continuous stream of tweets inside the onStatus() method. We are using the Kafka Producer code inside the onStatus() method to publish the tweets in Kafka. The following is the source code for the TwitterData class: public class TwitterData { /** The actual Twitter stream. It's set up to collect raw JSON data */ private TwitterStream twitterStream; static String consumerKeyStr = "r1wFskT3q"; static String consumerSecretStr = "fBbmp71HKbqalpizIwwwkBpKC"; static String accessTokenStr = "298FPfE16frABXMcRIn7aUSSnNneMEPrUuZ"; static String accessTokenSecretStr = "1LMNZZIfrAimpD004QilV1pH3PYTvM"; public void start() { ConfigurationBuilder cb = new ConfigurationBuilder(); cb.setOAuthConsumerKey(consumerKeyStr); cb.setOAuthConsumerSecret(consumerSecretStr); cb.setOAuthAccessToken(accessTokenStr); cb.setOAuthAccessTokenSecret(accessTokenSecretStr); cb.setJSONStoreEnabled(true); cb.setIncludeEntitiesEnabled(true); // instance of TwitterStreamFactory twitterStream = new TwitterStreamFactory(cb.build()).getInstance(); final Producer<String, String> producer = new KafkaProducer<String, String>(getProducerConfig());// topicDetails CreateTopic("127.0.0.1:2181").createTopic("twitterData", 2, 1); /** Twitter listener **/ StatusListener listener = new StatusListener() { public void onStatus(Status status) { ProducerRecord<String, String> data = new ProducerRecord<String, String>("twitterData", DataObjectFactory.getRawJSON(status)); // send the data to kafka producer.send(data); } public void onException(Exception arg0) { System.out.println(arg0); } arg0) {  }; public void onDeletionNotice(StatusDeletionNotice } public void onScrubGeo(long arg0, long arg1) { } public void onStallWarning(StallWarning arg0) { } public void onTrackLimitationNotice(int arg0) { } /** Bind the listener **/ twitterStream.addListener(listener); /** GOGOGO **/ twitterStream.sample(); } private Properties getProducerConfig() { Properties props = new Properties(); // List of kafka borkers. Complete list of brokers is not required as // the producer will auto discover the rest of the brokers. props.put("bootstrap.servers", "localhost:9092"); props.put("batch.size", 1); // new sending // Serializer used for sending data to kafka. Since we are // string, // we are using StringSerializer. props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer"); props.put("producer.type", "sync"); return props; } public static void main(String[] args) throws InterruptedException { new TwitterData().start(); } Use valid Kafka properties before executing the TwitterData. After executing the preceding class, the user will have a real-time stream of Twitter tweets in Kafka. In the next section, we are going to cover how we can use Storm to calculate the sentiments of the collected tweets. To summarize we covered how to install a single node Apache Kafka cluster and how to collect tweet from Twitter to store in a Kafka cluster If you enjoyed this post, check out the book Mastering Apache Storm to know more about different types of real time processing techniques used to create distributed applications.
Read more
  • 0
  • 0
  • 6101

article-image-lambda-functions
Packt
05 Jul 2017
16 min read
Save for later

Lambda Functions

Packt
05 Jul 2017
16 min read
In this article, by Udita Gupta and Yohan Wadia, the authors of the book Mastering AWS Lambda, we are going to take things a step further by learning the anatomy of a typical Lambda Function and also how to actually write your own functions. We will cover the programming model for a Lambda function using simple functions as examples, the use of logs and exceptions and error handling. (For more resources related to this topic, see here.) The Lambda programming model Certain applications can be broken down into one or more simple nuggets of code called as functions and uploaded to AWS Lambda for execution. Lambda then takes care of provisioning the necessary resources to run your function along with other management activities such as auto-scaling of your functions, their availability, and so on. So what exactly are we supposed to do in all this? A developer basically has three tasks to perform when it comes to working with Lambda: Writing the code Packaging it for deployment Finally monitoring its execution and fine tuning In this section, we are going to explore the different components that actually make up a Lambda Function by understanding what AWS calls as a programming model or a programming pattern. As of date, AWS officially supports Node.js, Java, Python, and C# as the programming languages for writing Lambda functions, with each language following a generic programming pattern that comprises of certain concepts which we will see in the following sections. Handler The handler function is basically a function that Lambda calls first for execution. A handler function is capable of processing incoming event data that is passed to it as well as invoking other functions or methods from your code. We will be concentrating a lot of our code and development on Node.js; however, the programming model remains more or less the same for the other supported languages as well. A skeleton structure of a handler function is shown as follows: exports.myHandler = function(event, context, callback) { // Your code goes here. callback(); } Where, myHandler is the name of your handler function. By exporting it we make sure that Lambda knows which function it has to invoke first. The other parameters that are passed with the handler function are: event: Lambda uses this parameter to pass any event related data back to the handler. context: Lambda again uses this parameter to provide the handler with the function's runtime information such as the name of the function, the time it took to execute, and so on . callback: This parameter is used to return any data back to its caller. The callback parameter is the only optional parameter that gets passed when writing handlers. If not specified, AWS Lambda will call it implicitly and return the value as null. The callback parameter also supports two optional parameters in the form of error and result where error will return any of the function's error information back to the caller while result will return any result of your function's successful execution. Here are a few simple examples of invoking callbacks in your handler: callback() callback(null, 'Hello from Lambda') callback(error) The callback parameter is supported only in Node.js runtime v4.3. You will have to use the context methods in case your code supports earlier Node.js runtime (v0.10.42) Let us try out a simple handler example with a code: exports.myHandler = function(event, context, callback) { console.log("value = " + event.key); console.log("functionName = ", context.functionName); callback(null, "Yippee! Something worked!"); }; The following code snippet will print the value of an event (key) that we will pass to the function, print the function's name as part of the context object and finally print the success message Yippee! Something worked! if all goes well! Login to the AWS Management Console and select AWS Lambda from the dashboard. Select the Create a Lambda function option. From the Select blueprint page, select the Blank Function blueprint. Since we are not configuring any triggers for now, simple click on Next at the Configure triggers page. Provide a suitable Name and Description for your Lambda function and paste the preceding code snippet in the inline code editor as shown: Next, in the Lambda function handler and role section on the same page, type in the correct name of your Handler as shown. The handler name should match with the handler name in your function to work. Remember also to select the basic-lambda-role for your function's execution before selecting the Next button: In the Review page, select the Create function option. With your function now created, select the Test option to pass the sample event to our function. In the Sample event, pass the following event and select the Save and test option: { "key": "My Printed Value!!" } With your code execution completed, you should get a similar execution result as shown in the following figure. The important things to note here are the values for the event, context and callback parameters. You can note the callback message being returned back to the caller as the function executed successfully. The other event and context object values are printed in the Log output section as highlighted in the following figure: In case you end up with any errors, make sure the handler function name matches the handler name that you passed during the function's configuration. Context object The context object is a really useful utility when it comes to obtaining runtime information about your function. The context object can provide information such as the executing function's name, the time remaining before Lambda terminates your function's execution, the log name and stream associated with your function and much more. The context object also comes with its own methods that you can call to correctly terminate your function's executions such as context.succed(), context.fail(), context.done(), and so on. However, post April 2016, Lambda has transitioned the Node.js runtime from v0.10.42 to v4.3 which does support these methods however encourages to use the callback() for performing the same actions. Here are some of the commonly used context object methods and properties described as follows: getRemainingTimeInMillis(): This property returns the number of milliseconds left for execution before Lambda terminates your function. This comes in really handy when you want to perform some corrective actions before your function exits or gets timed out. callbackWaitsForEmptyEventLoop: This property is used to override the default behaviour of a callback() function, such as to wait till the entire event loop is processed and only then return back to the caller. If set to false, this property causes the callback() function to stop any further processing in the event loop even if there are any other tasks to be performed. The default value is set to true. functionName: This property returns the name of the executing Lambda function. functionVersion: The current version of the executing Lambda function. memoryLimitInMB: The amount of resource in terms of memory set for your Lambda function. logGroupName: This property returns the name of the CloudWatch Log Group that stores function's execution logs. logStreamName: This property returns the name of the CloudWatch Log Stream that stores function's execution logs. awsRequestID: This property returns the request ID associated with that particular function's execution. If you are using Lambda functions as mobile backend processing services, you can then extract additional information about your mobile application using the context of identity and clientContext objects. These are invoked using the AWS Mobile SDK. To learn more, click here http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-context.html. Let us look at a simple example to understand the context object a bit better. In this example, we are using the context object callbackWaitsForEmptyEventLoop and demonstrating its working by setting the object's value to either yes or no on invocation: Login to the AWS Management Console and select AWS Lambda from the dashboard. Select the Create a Lambda function option. From the Select blueprint page, select the Blank Function blueprint. Since we are not configuring any triggers for now, simple click on Next at the Configure triggers page. Provide a suitable Name and Description for your Lambda function and paste the following code in the inline code editor: exports.myHandler = (event, context, callback) => { console.log('remaining time =', context.getRemainingTimeInMillis()); console.log('functionName =', context.functionName); console.log('AWSrequestID =', context.awsRequestId); console.log('logGroupName =', context.logGroupName); console.log('logStreamName =', context.logStreamName); switch (event.contextCallbackOption) { case "no": setTimeout(function(){ console.log("I am back from my timeout of 30 seconds!!"); },30000); // 30 seconds break break; case "yes": console.log("The callback won't wait for the setTimeout() n if the callbackWaitsForEmptyEventLoop is set to false"); setTimeout(function(){ console.log("I am back from my timeout of 30 seconds!!"); },30000); // 30 seconds break context.callbackWaitsForEmptyEventLoop = false; break; default: console.log("The Default code block"); } callback(null, 'Hello from Lambda'); }; Next, in the Lambda function handler and role section on the same page, type in the correct name of your Handler as shown. The handler name should match with the handler name in your function to work. Remember also to select the basic-lambda-role for your function's execution. The final change that we will do is change the Timeout value of our function from the default 3 seconds to 1 minute specifically for this example. Click Next to continue: In the Review page, select the Create function option. With your function now created, select the Test option to pass the sample event to our function. In the Sample event, pass the following event and select the Save and test option. You should see a similar output in the Log output window as shown: With the contextCallbackOption set to yes, the function does not wait for the 30 seconds setTimeout() function and will exit, however it prints the function's runtime information such as the remaining execution time, the function name, and so on. Now set the contextCallbackOption to no and re-run the test and verify the output. This time, you can see the setTimeout() function getting called and verify the same by comparing the remaining time left for execution with the earlier test run.   Logging You can always log your code's execution and activities using simple log statements. The following statements are supported for logging with Node.js runtime: console.log() console.error() console.warn() console.info() The logs can be viewed using both the Management Console as well as the CLI. Let us quickly explore both the options. Using the Management Console We have already been using Lambda's dashboard to view the function's execution logs, however the logs are only for the current execution. To view your function's logs from the past, you need to view them using the CloudWatch Logs section: To do so, search and select CloudWatch option from the AWS Management Console. Next, select the Logs option to display the function's logs as shown in the following figure: You can use the Filter option to filter out your Lambda logs by typing in the log group name prefix as /aws/lambda.   Select any of the present Log Groups and its corresponding Log Stream Name to view the complete and detailed execution logs of your function. If you do not see any Lambda logs listed out here it is mostly due to your Lambda execution role. Make sure your role has the necessary access rights to create the log group and log stream along with the capability to put log events. Using the CLI The CLI provides two ways using which you can view your function's execution logs: The first is using the Lambda function's invoke command itself. The invoke command when used with the --log-type parameter will print the latest 4 KB of log data that is written to CloudWatch Logs. To do so, first list out all available functions in your current region using the following command: # aws lambda list-functions Next, pick a Lambda function that you wish to invoke and substitute that function's name and payload with the following example snippet: # aws lambda invoke --invocation-type RequestResponse --function-name myFirstFunction --log-type Tail --payload '{"key1":"Lambda","key2":"is","key3":"awesome!"}' output.txt The second way is by using a combination of the context() object and the CloudWatch CLI. You can obtain your function's log group name and the log stream name using the context.logGroupName and the context.logStreamName. Next, substitute the data gathered from the output of these parameters in the following command: # aws logs get-log-events --log-group-name "/aws/lambda/myFirstFunction" --log-stream-name "2017/02/07/[$LATEST]1ae6ac9c77384794a3202802c683179a" If you run into the error The specified log stream does not exist in spite of providing correct values for the log group name and stream name; then make sure to add the "" escape character in the [$LATEST] as shown. Let us look at a few options that you can additionally pass with the get-log-events command: --start-time: The start of the log's time range. All times are in UTC. --end-time: The end of the log's time range. All times are in UTC. --next-token: The token for the next set of items to return. (You received this token from a previous call.) --limit: Used to set the maximum number of log events returned. By default the limit is set to either 10,000 log events. Alternatively, if you don't wish to use the context() objects in your code, you can still filter out the log group name and log stream name by using a combination of the following commands: # aws logs describe-log-groups --log-group-name-prefix "/aws/lambda/" The describe-log-groups command will list all the log groups that are prefixed with /aws/lambda. Make a note of your function's log group name from this output. Next, execute the following command to list your log group name's associated log stream names: # aws logs describe-log-streams --log-group-name "/aws/lambda/myFirstFunction" Make a note of the log stream name and substitute the same in the next and final command to view your log events for that particular log stream name: # aws logs get-log-events --log-group-name "/aws/lambda/myFirstFunction" --log-stream-name "2017/02/07/[$LATEST]1ae6ac9c77384794a3202802c683179a" Once again, make sure to add the backslash "" in the [$LATEST] to avoid the The specified log stream does not exist error. With the logging done, let's move on to the next piece of the programming model called exceptions. Exceptions and error handling Functions have the ability to notify AWS Lambda in case it failed to execute correctly. This is primarily done by the function passing the error object to Lambda which converts the same to a string and returns it to the user as an error message. The error messages that are returned also depend on the invocation type of the function; for example, if your function performs a synchronous execution (RequestResponse invocation type), then the error is returned back to the user and displayed on the Management Console as well as in the CloudWatch Logs. For any asynchronous executions (event invocation type), Lambda will not return anything. Instead it logs the error messages to CloudWatch Logs. Let us examine a function's error and exception handling capabilities with a simple example of a calculator function that accepts two numbers and an operand as the test events during invocation: Login to the AWS Management Console and select AWS Lambda from the dashboard. Select the Create a Lambda function option. From the Select blueprint page, select the Blank Function blueprint. Since we are not configuring any triggers for now, simple click on Next at the Configure triggers page. Provide a suitable Name and Description for your Lambda function and paste the following code in the inline code editor: exports.myHandler = (event, context, callback) => { console.log("Hello, Starting the "+ context.functionName +" Lambda Function"); console.log("The event we pass will have two numbers and an operand value"); // operand can be +, -, /, *, add, sub, mul, div console.log('Received event:', JSON.stringify(event, null, 2)); var error, result; if (isNaN(event.num1) || isNaN(event.num2)) { console.error("Invalid Numbers"); // different logging error = new Error("Invalid Numbers!"); // Exception Handling callback(error); } switch(event.operand) { case "+": case "add": result = event.num1 + event.num2; break; case "-": case "sub": result = event.num1 - event.num2; break; case "*": case "mul": result = event.num1 * event.num2; break; case "/": case "div": if(event.num2 === 0){ console.error("The divisor cannot be 0"); error = new Error("The divisor cannot be 0"); callback(error, null); } else{ result = event.num1/event.num2; } break; default: callback("Invalid Operand"); break; } console.log("The Result is: " + result); callback(null, result); }; Next, in the Lambda function handler and role section on the same page, type in the correct name of your Handler. The handler name should match with the handler name in your function to work. Remember also to select the basic-lambda-role for your function's execution. Leave the rest of the values to their defaults and click Next to continue. In the Review page, select the Create function option. With your function now created, select the Test option to pass the sample event to our function. In the Sample event, pass the following event and select the Save and test option. You should see a similar output in the Log output window as shown: { "num1": 3, "num2": 0, "operand": "div" } So what just happened there? Well first, we can print simple user friendly error messages with the help of the console.error() statement. Additionally, we can also print the stackTrace array of the error by passing the error in the callback() as shown: error = new Error("The divisor cannot be 0"); callback(error, null); You can also view the custom error message and the stackTrace JSON array both from the Lambda dashboard as well as from the CloudWatch Logs section. Next, give this code a couple of tries with some different permutations and combinations of events and check out the results. You can even write your own custom error messages and error handlers that can perform some additional task when an error is returned by the function. With this we come towards the end of a function's generic programming model and its components.  Summary We deep dived into the Lambda programming model and understood each of its sub components (handlers, context objects, errors and exceptions) with easy to follow examples.
Read more
  • 0
  • 0
  • 6095

article-image-puppet-server-and-agents
Packt
16 Aug 2017
18 min read
Save for later

Puppet Server and Agents

Packt
16 Aug 2017
18 min read
In this article by Martin Alfke, the author of the book Puppet Essentials - Third Edition, we will following topics: The Puppet server Setting up the Puppet Agent (For more resources related to this topic, see here.) The Puppetserver Many Puppet-based workflows are centered on the server, which is the central source of configuration data and authority. The server hands instructions to all the computer systems in the infrastructure (where agents are installed). It serves multiple purposes in the distributed system of Puppet components. The server will perform the following tasks: Storing manifests and compiling catalogs Serving as the SSL certification authority Processing reports from the agent machines Gathering and storing information about the agents As such, the security of your server machine is paramount. The requirements for hardening are comparable to those of a Kerberos Key Distribution Center. During its first initialization, the Puppet server generates the CA certificate. This self-signed certificate will be distributed among and trusted by all the components of your infrastructure. This is why its private key must be protected very carefully. New agent machines request individual certificates, which are signed with the CA certificate. The terminology around the master software might be a little confusing. That's because both the terms, Puppet Master and Puppet Server, are floating around, and they are closely related too. Let's consider some technological background in order to give you a better understanding of what is what. Puppet's master service mainly comprises a RESTful HTTP API. Agents initiate the HTTPS transactions, with both sides identifying each other using trusted SSL certificates. During the time when Puppet 3 and older versions were current, the HTTPS layer was typically handled by Apache. Puppet's Ruby core was invoked through the Passenger module. This approach offered good stability and scalability. Puppet Inc. has improved upon this standard solution with a specialized software called puppetserver. The Ruby-based core of the master remains basically unchanged, although it now runs on JRuby instead of Ruby's own MRI. The HTTPS layer is run by Jetty, sharing the same Java Virtual Machine with the master. By cutting out some middlemen, puppetserver is faster and more scalable than a Passenger solution. It is also significantly easier to set up. Setting up the server machine Getting the puppetserver software onto a Linux machine is just as simple as the agent package. Packages are available on Red Hat Enterprise Linux and its derivatives,derivatives, Debian and Ubuntu and any other operating system which is supported to run a Puppet server. Until now the Puppet server must run on a Linux based Operating System and can not run on Windows or any other U*ix.U*ix. A great way to get Puppet Inc. packages on any platform is the Puppet Collection. Shortly after the release of Puppet 4, Puppet Inc. created this new way of supplying software. This can be considered as a distribution in its own right. Unlike Linux distributions, it does not contain a Kernel, system tools, and libraries. Instead, it comprises various software from the Puppet ecosystem. Software versions that are available from the same Puppet Collection are guaranteed to work well together. Use the following commands to install puppetserver from the first Puppet Collection (PC1) on a Debian 7 machine. (The Collection for Debian 8 has not yet received a puppetserver package at the time of writing this.) root@puppetmaster# wget http://apt.puppetlabs.com/puppetlabs-release-pc1-jessie.debhttp://apt.puppetlabs.com/puppetlabs-release-pc1-jessie.deb root@puppetmaster# dpkg -i puppetlabs-release-pc1-jessie.debpuppetlabs-release-pc1-jessie.deb root@puppetmaster# apt-get update root@puppetmaster# apt-get install puppetserver The puppetserver package comprises only the Jetty server and the Clojure API, but the all-in-one puppet-agent package is pulled in as a dependency. The package name, puppet-agent, is misleading. This AIO package contains all the parts of Puppet including the master core, a vendored Ruby build, and the several pieces of additional software. Specifically, you can use the puppet command on the master node. You will soon learn how this is useful. However, when using the packages from Puppet Labs, everything gets installed under /opt/puppetlabs. It is advisable to make sure that your PATH variable always includes the /opt/puppetlabs/bin directory so that the puppet command is found here. Regardless of this, once the puppetserver package is installed, you can start the master service: root@puppetmaster# systemctl start puppetserver Depending on the power of your machine, the startup can take a few minutes. Once initialization completes, the server will operate very smoothly though. As soon as the master port 8140 is open, your Puppet master is ready to serve requests. If the service fails to start, there might be an issue with certificate generation. (We observed such issues with some versions of the software.) Check the log file at /var/log/puppetlabs/puppetserver/puppetserver-daemon.log. If it indicates that there are problems while looking up its certificate file, you can work around the problem by temporarily running a standalone master as follows: puppet master –-no-daemonize. After initialization, you can stop this process. The certificate is available now, and puppetserver should now be able to start as well. Another reason for start failures is insufficient amount of memory. The Puppet server process needs 2 GB of memory. Creating the master manifest The master compiles manifests for many machines, but the agent does not get to choose which source file is to be used—this is completely at the master's discretion. The starting point for any compilation by the master is always the site manifest, which can be found in /opt/puppetlabs/code/environments/production/manifests/. Each connecting agent will use all the manifests found here. Of course, you don't want to manage only one identical set of resources on all your machines. To define a piece of manifest exclusively for a specific agent, put it in a node block. This block's contents will only be considered when the calling agent has a matching common name in its SSL certificate. You can dedicate a piece of the manifest to a machine with the name of agent, for example: node 'agent' { $packages = [ 'apache2', 'libapache2-mod-php5', 'libapache2-mod-passenger', ] package { $packages: ensure => 'installed', before => Service['apache2'], } service { 'apache2': ensure => 'running', enable => true, } } Before you set up and connect your first agent to the master, step back and think about how the master should be addressed. By default, agents will try to resolve the unqualified puppet hostname in order to get the master's address. If you have a default domain that is being searched by your machines, you can use this as a default and add a record for puppet as a subdomain (such as puppet.example.net). Otherwise, pick a domain name that seems fitting to you, such as master.example.net or adm01.example.net. What's important is the following: All your agent machines can resolve the name to an address The master process is listening for connections on that address The master uses a certificate with the chosen name as CN or DNS Alt Names The mode of resolution depends on your circumstances—the hosts file on each machine is one ubiquitous possibility. The Puppet server listens on all the available addresses by default. This leaves the task of creating a suitable certificate, which is simple. Configure the master to use the appropriate certificate name and restart the service. If the certificate does not exist yet, Puppet will take the necessary steps to create it. Put the following setting into your /etc/puppetlabs/puppet/puppet.conf file on the master machine: [main] [main] certname=puppetmaster.example.net In Puppet versions before 4.0, the default location for the configuration file is /etc/puppet/puppet.conf. Upon its next start, the master will use the appropriate certificate for all SSL connections. The automatic proliferation of SSL data is not dangerous even in an existing setup, except for the certification authority. If the master were to generate a new CA certificate at any point in time, it would break the trust of all existing agents. Make sure that the CA data is neither lost nor compromised. All previously signed certificates become obsolete whenever Puppet needs to create a new certification authority. The default storage location is /etc/puppetlabs/puppet/ssl/ca for Puppet 4.0 and higher, and /var/lib/puppet/ssl/ca for older versions. Inspecting the configuration settings All the customization of the master's parameters can be made in the puppet.conf file. The operating system packages ship with some settings that are deemed sensible by the respective maintainers. Apart from these explicit settings, Puppet relies on defaults that are either built-in or derived from the : root@puppetmaster # puppet master --configprint manifest /etc/puppetlabs/code/environments/production/manifests Most users will want to rely on these defaults for as many settings as possible. This is possible without any drawbacks because Puppet makes all settings fully transparent using the --configprint parameter. For example, you can find out where the master manifest files are located. To get an overview of all available settings and their values, use the following command: root@puppetmaster# puppet master --configprint all | less While this command is especially useful on the master side, the same introspection is available for puppet apply and puppet agent.  Setting specific configuration entries is possible with the puppet config command: root@puppetmaster # puppet config set –-section main certname puppetmaster.example.net Setting up the Puppet agent As was explained earlier, the master mainly serves instructions to agents in the form of catalogs that are compiled from the manifest. You have also prepared a node block for your first agent in the master manifest. The plain Puppet package that allows you to apply a local manifest contains all the required parts in order to operate a proper agent. If you are using Puppet Labs packages, you need not install the puppetserver package. Just get puppet-agent instead. After a successful package installation, one needs to specify where Puppet agent can find the puppetserver: root@puppetmaster # puppet config set –-section agent server puppetmaster.example.net Afterwards the following invocation is sufficient for an initial test: root@agent# puppet agent --test Info: Creating a new SSL key for agent Error: Could not request certificate: getaddrinfo: Name or service not known Exiting; failed to retrieve certificate and waitforcert is disabled Puppet first created a new SSL certificate key for itself. For its own name, it picked agent, which is the machine's hostname. That's fine for now. An error occurred because the puppet name cannot be currently resolved to anything. Add this to /etc/hosts so that Puppet can contact the master: root@agent# puppet agent --test Info: Caching certificate for ca Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml Info: Creating a new SSL certificate request for agent Info: Certificate Request fingerprint (SHA256): 52:65:AE:24:5E:2A:C6:17:E2:5D:0A:C9: 86:E3:52:44:A2:EC:55:AE:3D:40:A9:F6:E1:28:31:50:FC:8E:80:69 Error: Could not request certificate: Error 500 on SERVER: Internal Server Error: java.io.FileNotFoundException: /etc/puppetlabs/puppet/ssl/ca/requests/agent.pem (Permission denied) Exiting; failed to retrieve certificate and waitforcert is disabled Note how Puppet conveniently downloaded and cached the CA certificate. The agent will establish trust based on this certificate from now on. Puppet created a certificate request and sent it to the master. It then immediately tried to download the signed certificate. This is expected to fail—the master won't just sign a certificate for any request it receives. This behavior is important for proper security. There is a configuration setting that enables such automatic signing, but users are generally discouraged from using this setting because it allows the creation of arbitrary numbers of signed (and therefore, trusted) certificates to any user who has network access to the master. To authorize the agent, look for the CSR on the master using the puppet cert command: root@puppetmaster# puppet cert --list "agent" (SHA256) 52:65:AE:24:5E:2A:C6:17:E2:5D:0A:C9:86:E3:52:44:A2:EC:55:AE: 3D:40:A9:F6:E1:28:31:50:FC:8E:80:69 This looks alright, so now you can sign a new certificate for the agent: root@puppetmaster# puppet cert --sign agent Notice: Signed certificate request for agent Notice: Removing file Puppet::SSL::CertificateRequest agent at '/etc/puppetlabs/ puppet/ssl/ca/requests/agent.pem' When choosing the action for puppet cert, the dashes in front of the option name can be omitted—you can just use puppet cert list and puppet cert sign. Now the agent can receive its certificate for its catalog run as follows: root@agent# puppet agent --test Info: Caching certificate for agent Info: Caching certificate_revocation_list for ca Info: Caching certificate for agent Info: Retrieving pluginfacts Info: Retrieving plugin Info: Caching catalog for agent Info: Applying configuration version '1437065761' Notice: Applied catalog in 0.11 seconds The agent is now fully operational. It received a catalog and applied all resources within. Before you read on to learn how the agent usually operates, there is a note that is especially important for the users of Puppet 3. Since this is the common name in the master's certificate, the preceding command will not even work with a Puppet 3.x master. It works with puppetserver and Puppet 4 because the default puppet name is now included in the certificate's Subject Alternative Names by default. It is tidier to not rely on this alias name, though. After all, in production, you will probably want to make sure that the master has a fully qualified name that can be resolved, at least inside your network. You should therefore, add the following to the main section of puppet.conf on each agent machine: [agent] [agent] server=master.example.net In the absence of DNS to resolve this name, your agent will need an appropriate entry in its hosts file or a similar alternative way of address resolution. These steps are necessary in a Puppet 3.x setup. If you have been following along with a Puppet 4 agent, you might notice that after this change, it generates a new Certificate Signing Request: root@agent# puppet agent –test Info: Creating a new SSL key for agent.example.net Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml Info: Creating a new SSL certificate request for agent.example.net Info: Certificate Request fingerprint (SHA256): 85:AC:3E:D7:6E:16:62:BD:28:15:B6:18: 12:8E:5D:1C:4E:DE:DF:C3:4E:8F:3E:20:78:1B:79:47:AE:36:98:FD Exiting; no certificate found and waitforcert is disabled If this happens, you will have to use puppet cert sign on the master again.  The agent will then retrieve a new certificate. The agent's life cycle In a Puppet-centric workflow, you typically want all changes to the configuration of servers (perhaps even workstations) to originate on the Puppet master and propagate to the agents automatically. Each new machine gets integrated into the Puppet infrastructure with the master at its center and gets removed during the decommissioning, as shown in the following diagram: Insert image 6566_02_01.png The very first step—generating a key and a certificate signing request—is always performed implicitly and automatically at the start of an agent run if no local SSL data exists yet. Puppet creates the required data if no appropriate files are found. There will be a short description on how to trigger this behavior manually later in this section. The next step is usually the signing of the agent's certificate, which is performed on the master. It is a good practice to monitor the pending requests by listing them on the console: root@puppetmaster# puppet cert list oot@puppetmaster# puppet cert sign '<agent fqdn>' From this point on, the agent will periodically check with the master to load updated catalogs. The default interval for this is 30 minutes. The agent will perform a run of a catalog each time and check the sync state of all the contained resources. The run is performed for unchanged catalogs as well, because the sync states can change between runs. Before you manage to sign the certificate, the agent process will query the master in short intervals for a while. This can avoid a 30 minute delay if the certificate is not ready, right when the agent starts up. Launching this background process can be done manually through a simple command: root@agent# puppet agent However, it is preferable to do this through the puppet system service. When an agent machine is taken out of active service, its certificate should be invalidated. As is customary with SSL, this is done through revocation. The master adds the serial number of the certificate to its certificate revocation list. This list, too, is shared with each agent machine. Revocation is initiated on the master through the puppet cert command: root@puppetmaster# puppet cert revoke agent The updated CRL is not honored until the master service is restarted. If security is a concern, this step must not be postponed. The agent can then no longer use its old certificate: root@agent# puppet agent --test Warning: Unable to fetch my node definition, but the agent run will continue: Warning: SSL_connect SYSCALL returned=5 errno=0 state=unknown state [...] Error: Could not retrieve catalog from remote server: SSL_connect SYSCALL returned=5 errno=0 state=unknown state [...] Renewing an agent's certificate Sometimes, it is necessary during an agent machine's life cycle to regenerate its certificate and related data—the reasons can include data loss, human error, or certificate expiration, among others. Performing the regeneration is quite simple: all relevant files are kept at /etc/puppetlabs/puppet/ssl (for Puppet 3.x, this is /var/lib/puppet/ssl) on the agent machine. Once these files are removed (or rather, the whole ssl/ directory tree), Puppet will renew everything on the next agent run. Of course, a new certificate must be signed. This requires some preparation—just initiating the request from the agent will fail: root@agent# puppet agent –test Info: Creating a new SSL key for agent Info: Caching certificate for ca Info: Caching certificate for agent.example.net Error: Could not request certificate: The certificate retrievedfrom the master does not match the agent's private key. Certificate fingerprint: 6A:9F:12:C8:75:C0:B6:10:45:ED:C3:97:24:CC:98:F2:B6:1A:B5: 4C:E3:98:96:4F:DA:CD:5B:59:E0:7F:F5:E6 The master still has the old certificate cached. This is a simple protection against the impersonation of your agents by unauthorized entities. To fix this, remove the certificate from both the master and the agent and then start a Puppet run, which will automatically regenerate a certificate. On the master: puppet cert clean agent.example.net On the agent: On most platforms: find /etc/puppetlabs/puppet/ssl -name agent.example.net.pem –delete On Windows: del "/etc/puppetlabs/puppet/ssl/agent.example.net.pem" /f puppet agent –t Exiting; failed to retrieve certificate and waitforcert is disabled Once you perform the cleanup operation on the master, as advised in the preceding output, and remove the indicated file from the agent machine, the agent will be able to successfully place its new CSR: root@puppetmaster# puppet cert clean agent Notice: Revoked certificate with serial 18 Notice: Removing file Puppet::SSL::Certificate agent at '/etc/puppetlabs/ puppet/ssl/ca/signed/agent.pem' Removing file Puppet::SSL::Certificate agent at '/etc/puppetlabs/ puppet/ssl/certs/agent.pem'. The rest of the process is identical to the original certificate creation. The agent uploads its CSR to the master, where the certificate is created through the puppet cert sign command. Running the agent from cron There is an alternative way to operate the agent. We covered starting one long-running puppet agent process that does its work in set intervals and then goes back to sleep. However, it is also possible to have cron launch a discrete agent process in the same interval. This agent will contact the master once, run the received catalog, and then terminate. This has several advantages as follows: The agent operating system saves some resources The interval is precise and not subject to skew (when running the background agent, deviations result from the time that elapses during the catalog run), and distributed interval skew can lead to thundering herd effects Any agent crash or an inadvertent termination is not fatal Setting Puppet to run the agent from cron is also very easy to do—with Puppet! You can use a manifest such as the following: service { 'puppet': enable => false } cron { 'puppet-agent-run': user => 'root', command => 'puppet agent --no-daemonize --onetime --logdest=syslog', minute => fqdn_rand(60), hour => absent, } The fqdn_rand function computes a distinct minute for each of your agents. Setting the hour property to absent means that the job should run every hour. Summary In this article we have learned about the Puppet server and also learned how to setting up the Puppet agent.  Resources for Article: Further resources on this subject: Quick start – Using the core Puppet resource types [article] Understanding the Puppet Resources [article] Puppet: Integrating External Tools [article]
Read more
  • 0
  • 0
  • 6091
article-image-introducing-object-oriented-programmng-typescript
Packt
17 Feb 2016
13 min read
Save for later

Introducing Object Oriented Programmng with TypeScript

Packt
17 Feb 2016
13 min read
In this article, we will see how to group our functions in reusable components, such as classes or modules. This article will cover the following topics: SOLID principles Classes Association, aggregation, and composition Inheritance (For more resources related to this topic, see here.) SOLID principles In the early days of software development, developers used to write code with procedural programming languages. In procedural programming languages, the programs follow a top-to-bottom approach and the logic is wrapped with functions. New styles of computer programming, such as modular programming or structured programming, emerged when developers realized that procedural computer programs could not provide them with the desired level of abstraction, maintainability, and reusability. The development community created a series of recommended practices and design patterns to improve the level of abstraction and reusability of procedural programming languages, but some of these guidelines required a certain level of expertise. In order to facilitate adherence to these guidelines, a new style of computer programming known as object-oriented programming (OOP) was created. Developers quickly noticed some common OOP mistakes and came up with five rules that every OOP developer should follow to create a system that is easy to maintain and extend over time. These five rules are known as the SOLID principles. SOLID is an acronym introduced by Michael Feathers, which stands for the following principles: Single responsibility principle (SRP): This principle states that a software component (function, class, or module) should focus on one unique task (have only one responsibility). Open/closed principle (OCP): This principle states that software entities should be designed with application growth (new code) in mind (should be open to extension), but the application growth should require the fewer possible number of changes to the existing code (be closed for modification). Liskov substitution principle (LSP): This principle states that we should be able to replace a class in a program with another class as long as both classes implement the same interface. After replacing the class, no other changes should be required, and the program should continue to work as it did originally. Interface segregation principle (ISP): This principle states that we should split interfaces that are very large (general-purpose interfaces) into smaller and more specific ones (many client-specific interfaces) so that clients will only need to know about the methods that are of interest to them. Dependency inversion principle (DIP): This principle states that entities should depend on abstractions (interfaces) as opposed to depending on concretion (classes). In this article, we will see how to write TypeScript code that adheres to these principles so that our applications are easy to maintain and extend over time. Classes In this section, we will look at some details and OOP concepts through examples. Let's start by declaring a simple class: class Person { public name : string; public surname : string; public email : string; constructor(name : string, surname : string, email : string){ this.email = email; this.name = name; this.surname = surname; } greet() { alert("Hi!"); } } var me : Person = new Person("Remo", "Jansen", "[email protected]"); We use classes to represent the type of an object or entity. A class is composed of a name, attributes, and methods. The class in the preceding example is named Person and contains three attributes or properties (name, surname, and email) and two methods (constructor and greet). Class attributes are used to describe the object's characteristics, while class methods are used to describe its behavior. A constructor is a special method used by the new keyword to create instances (also known as objects) of our class. We have declared a variable named me, which holds an instance of the Person class. The new keyword uses the Person class's constructor to return an object whose type is Person. A class should adhere to the single responsibility principle (SRP). The Person class in the preceding example represents a person, including all their characteristics (attributes) and behaviors (methods). Now let's add some email as validation logic: class Person { public name : string; public surname : string; public email : string; constructor(name : string, surname : string, email : string) { this.surname = surname; this.name = name; if(this.validateEmail(email)) { this.email = email; } else { throw new Error("Invalid email!"); } } validateEmail() { var re = /S+@S+.S+/; return re.test(this.email); } greet() { alert("Hi! I'm " + this.name + ". You can reach me at " + this.email); } } When an object doesn't follow the SRP and it knows too much (has too many properties) or does too much (has too many methods), we say that the object is a God object. The Person class here is a God object because we have added a method named validateEmail that is not really related to the Person class's behavior. Deciding which attributes and methods should or should not be part of a class is a relatively subjective decision. If we spend some time analyzing our options, we should be able to find a way to improve the design of our classes. We can refactor the Person class by declaring an Email class, responsible for e-mail validation, and use it as an attribute in the Person class: class Email { public email : string; constructor(email : string){ if(this.validateEmail(email)) { this.email = email; } else { throw new Error("Invalid email!"); } } validateEmail(email : string) { var re = /S+@S+.S+/; return re.test(email); } } Now that we have an Email class, we can remove the responsibility of validating the emails from the Person class and update its email attribute to use the type Email instead of string: class Person { public name : string; public surname : string; public email : Email; constructor(name : string, surname : string, email : Email){ this.email = email; this.name = name; this.surname = surname; } greet() { alert("Hi!"); } } Making sure that a class has a single responsibility makes it easier to see what it does and how we can extend/improve it. We can further improve our Person and Email classes by increasing the level of abstraction of our classes. For example, when we use the Email class, we don't really need to be aware of the existence of the validateEmail method; so this method could be invisible from outside the Email class. As a result, the Email class would be much simpler to understand. When we increase the level of abstraction of an object, we can say that we are encapsulating the object's data and behavior. Encapsulation is also known as information hiding. For example, the Email class allows us to use emails without having to worry about e-mail validation because the class will deal with it for us. We can make this clearer by using access modifiers (public or private) to flag as private all the class attributes and methods that we want to abstract from the use of the Email class: class Email { private email : string; constructor(email : string){ if(this.validateEmail(email)) { this.email = email; } else { throw new Error("Invalid email!"); } } private validateEmail(email : string) { var re = /S+@S+.S+/; return re.test(email); } get():string { return this.email; } } We can then simply use the Email class without needing to explicitly perform any kind of validation: var email = new Email("[email protected]"); Interfaces The feature that we will miss the most when developing large-scale web applications with JavaScript is probably interfaces. We have seen that following the SOLID principles can help us to improve the quality of our code, and writing good code is a must when working on a large project. The problem is that if we attempt to follow the SOLID principles with JavaScript, we will soon realize that without interfaces, we will never be able to write SOLID OOP code. Fortunately for us, TypeScript features interfaces. Traditionally, in OOP, we say that a class can extend another class and implement one or more interfaces. An interface can implement one or more interfaces and cannot extend another class or interface. Wikipedia's definition of interfaces in OOP is as follows: In object-oriented languages, the term interface is often used to define an abstract type that contains no data or code, but defines behaviors as method signatures. Implementing an interface can be understood as signing a contract. The interface is a contract, and when we sign it (implement it), we must follow its rules. The interface rules are the signatures of the methods and properties, and we must implement them. We will see many examples of interfaces later in this article. In TypeScript, interfaces don't strictly follow this definition. The two main differences are that in TypeScript: An interface can extend another interface or class An interface can define data and behaviors as opposed to only behaviors Association, aggregation, and composition In OOP, classes can have some kind of relationship with each other. Now, we will take a look at the three different types of relationships between classes. Association We call association those relationships whose objects have an independent lifecycle and where there is no ownership between the objects. Let's take an example of a teacher and student. Multiple students can associate with a single teacher, and a single student can associate with multiple teachers, but both have their own lifecycles (both can be create and delete independently); so when a teacher leaves the school, we don't need to delete any students, and when a student leaves the school, we don't need to delete any teachers. Aggregation We call aggregation those relationships whose objects have an independent lifecycle, but there is ownership, and child objects cannot belong to another parent object. Let's take an example of a cell phone and a cell phone battery. A single battery can belong to a phone, but if the phone stops working, and we delete it from our database, the phone battery will not be deleted because it may still be functional. So in aggregation, while there is ownership, objects have their own lifecycle. Composition We use the term composition to refer to relationships whose objects don't have an independent lifecycle, and if the parent object is deleted, all child objects will also be deleted. Let's take an example of the relationship between questions and answers. Single questions can have multiple answers, and answers cannot belong to multiple questions. If we delete questions, answers will automatically be deleted. Objects with a dependent life cycle (answers, in the example) are known as weak entities. Sometimes, it can be a complicated process to decide if we should use association, aggregation, or composition. This difficulty is caused in part because aggregation and composition are subsets of association, meaning they are specific cases of association. Inheritance One of the most fundamental object-oriented programming features is its capability to extend existing classes. This feature is known as inheritance and allows us to create a new class (child class) that inherits all the properties and methods from an existing class (parent class). Child classes can include additional properties and methods not available in the parent class. Let's return to our previously declared Person class. We will use the Person class as the parent class of a child class named Teacher: class Person { public name : string; public surname : string; public email : Email; constructor(name : string, surname : string, email : Email){ this.name = name; this.surname = surname; this.email = email; } greet() { alert("Hi!"); } } This example is included in the companion source code. Once we have a parent class in place, we can extend it by using the reserved keyword extends. In the following example, we declare a class called Teacher, which extends the previously defined Person class. This means that Teacher will inherit all the attributes and methods from its parent class: class Teacher extends Person { class Teacher extends Person { teach() { alert("Welcome to class!"); } } Note that we have also added a new method named teach to the class Teacher. If we create instances of the Person and Teacher classes, we will be able to see that both instances share the same attributes and methods with the exception of the teach method, which is only available for the instance of the Teacher class: var teacher = new Teacher("remo", "jansen", new Email("[email protected]")); var me = new Person("remo", "jansen", new Email("[email protected]")); me.greet(); teacher.greet(); me.teach(); // Error : Property 'teach' does not exist on type 'Person' teacher.teach(); Sometimes, we will need a child class to provide a specific implementation of a method that is already provided by its parent class. We can use the reserved keyword super for this purpose. Imagine that we want to add a new attribute to list the teacher's subjects, and we want to be able to initialize this attribute through the teacher constructor. We will use the super keyword to explicitly reference the parent class constructor inside the child class constructor. We can also use the super keyword when we want to extend an existing method, such as greet. This OOP language feature that allows a subclass or child class to provide a specific implementation of a method that is already provided by its parent classes is known as method overriding. class Teacher extends Person { public subjects : string[]; constructor(name : string, surname : string, email : Email, subjects : string[]){ super(name, surname, email); this.subjects = subjects; } greet() { super.greet(); alert("I teach " + this.subjects); } teach() { alert("Welcome to Maths class!"); } } var teacher = new Teacher("remo", "jansen", new Email("[email protected]"), ["math", "physics"]); We can declare a new class that inherits from a class that is already inheriting from another. In the following code snippet, we declare a class called SchoolPrincipal that extends the Teacher class, which extends the Person class: class SchoolPrincipal extends Teacher { manageTeachers() { alert("We need to help students to get better results!"); } } If we create an instance of the SchoolPrincipal class, we will be able to access all the properties and methods from its parent classes (SchoolPrincipal, Teacher, and Person): var principal = new SchoolPrincipal("remo", "jansen", new Email("[email protected]"), ["math", "physics"]); principal.greet(); principal.teach(); principal.manageTeachers(); It is not recommended to have too many levels in the inheritance tree. A class situated too deeply in the inheritance tree will be relatively complex to develop, test, and maintain. Unfortunately, we don't have a specific rule that we can follow when we are unsure whether we should increase the depth of the inheritance tree (DIT). We should use inheritance in such a way that it helps us to reduce the complexity of our application and not the opposite. We should try to keep the DIT between 0 and 4 because a value greater than 4 would compromise encapsulation and increase complexity. Summary In this article, we saw how to work with classes, and interfaces in depth. We were able to reduce the complexity of our application by using techniques such as encapsulation and inheritance. To learn more about TypeScript, the following books published by Packt Publishing (https://www.packtpub.com/) are recommended: TypeScript Essentials (https://www.packtpub.com/web-development/typescript-essentials) Mastering TypeScript(https://www.packtpub.com/web-development/mastering-typescript) Resources for Article: Further resources on this subject: Writing SOLID JavaScript code with TypeScript [article] Introduction to TypeScript [article] An Introduction to Mastering JavaScript Promises and Its Implementation in Angular.js [article]
Read more
  • 0
  • 0
  • 6090

article-image-optimizing-netscaler-traffic
Packt
14 Oct 2015
11 min read
Save for later

Optimizing NetScaler Traffic

Packt
14 Oct 2015
11 min read
In this article by Marius Sandbu, author of the book Implementing NetScaler VPX™ - Second Edition, explains the purpose of NetScaler is to act as a logistics department; it serves content to different endpoints using different protocols across different types of media, and it can either be a physical device or a device on top of a hypervisor within a private cloud infrastructure. Since there are many factors are at play here, there is room for tuning and improvement. Some of the topics we will go through in this article are as follows: Tuning for virtual environments Tuning TCP traffic (For more resources related to this topic, see here.) Tuning for virtual environments When setting up a NetScaler in a virtual environment, we need to keep in mind that there are many factors that influence how it will perform, for instance, the underlying CPU of the virtual host, NIC throughput and capabilities, vCPU over allocation, NIC teaming, MTU size, and so on. So, always important to remember the hardware requirements when setting up a NetScaler VPX on a virtualization host. Another important factor to keep in mind when setting up a NetScaler VPX is the concepts of Package Engines. By default, when we set up or import a NetScaler, it is set up with two vCPU. The first of these two is dedicated for management purpose and the second vCPU is dedicated to doing all the packet processing, such as content switching, SSL offloading, ICA-proxy, and so on. It is important to note that the second vCPU might be seen as 100% utilized in the hypervisor performance monitoring tools, but the correct way to check if it is being utilized is by using the CLI command stat system Now, by default, VPX 10 and VPX 200 only have support for one packet engine. This is because of the fact that due to its bandwidth limitations it does not require more packet engine CPUs to process the packets. On the other hand, VPX 1000 and VPX 3000 have support for up to 3 packet engines. This, in most cases, is needed to be able to process all the packets that are going through the system, if the bandwidth is going to be utilized to its fullest. In order to add a new packet engine, we need to assign more vCPUs to the VPX and more memory. Packet engines also have the benefit of load balancing processing between them, so instead of having a vCPU that is 100% utilized, we can even the load between multiple vCPUs and get an even better performance and bandwidth. The following chart shows the different editions and support for multiple packet engines: License/Memory 2 GB 4 GB 6 GB 8 GB 10 GB 12 GB VPX 10 1 1 1 1 1 1 VPX 200 1 1 1 1 1 1 VPX 1000 1 2 3 3 3 3 VPX 3000 1 2 3 3 3 3 It is important for us to remember that multiple PEs are only available for VMware, XenServer, and Hyper-V, but not on KVM. If we plan on using NIC-teaming on the underlying virtualization host, there are some important aspects to consider. Most of the different vendors have guidelines which describe the kinds of load balancing techniques that are available in the hypervisor. For instance, Microsoft has a guide that describes their features. You can find the guide at http://www.microsoft.com/en-us/download/details.aspx?id=30160. One of the NIC teaming options, called Switch Independent Dynamic Mode, has an interesting side effect; it replaces the source MAC address of the virtual machine with the one of the primary NIC on the host, hence we might experience packet loss on a VPX. Therefore, it is recommended in most cases that we have LACP/LAG, or in case of Hyper-V, use the HyperVPort distribution feature instead. Note that the features such as SRV-IO or PCI pass through are not supported for NetScaler VPX. NetScaler 11 also introduced the support for Jumbo Frames for the VPX, which allows for a much higher payload in an Ethernet frame. Instead of the traditional 1500 bytes, we can scale up to 9000 bytes of payload. This allows for a much lower overhead since the frames contain more data. This requires that the underlying NIC on the hypervisor supports this feature and is enabled as well, and this in most cases just work for communication with backend resources and not with users accessing public resources. This is because of the fact that most routers and ISP block such high MTU. This feature can be configured on at the Interface level in NetScaler, which can be done under System | Network | Interface then choose Select Interface and click on Edit. Here, we have an option called Maximum Transmission Unit which can be adjusted up to 9,216 Bytes. It is important to note that NetScaler communicates with backend resources using Jumbo frames, and then adjusts the MTU when communicating back with clients. It can also communicate with Jumbo frames in both paths, in case the NetScaler is set up as a backend load balancer. It is important to note that NetScaler only supports Jumbo frames load balancing for the following protocols: TCP TCP based protocols, like HTTP SIP RADIUS TCP tuning Much of the traffic which is going through NetScaler is based on the TCP protocol. Either it is ICA-Proxy, HTTP and so on. TCP is a protocol that provides reliable, error-checked delivery of packets back and forth. This ensures that data is successfully transferred before being processed further. TCP has many features to adjust bandwidth during transfer, congestion checking, adjusting segment sizing, and so on. We will delve a little into all these features in this section. We can adjust the way NetScaler uses TCP using TCP profiles, by default all services and vServers that are created on the NetScaler uses the default TCP profile nstcp_default_profile. Note that these profiles can be found by navigating to System | Profiles | TCP Profiles. Make sure not to alter the default TCP profile without properly consulting the network team, as this affects the way in which TCP works for all default services on the NetScaler. This default profile has most of the different TCP features turned off; this is to ensure compatibility with most infrastructures. The profile has not been adjusted much since it was first added in NetScaler. Citrix also has a lot of other different profiles depending on the use cases, so we are going to look a bit closer on the different options we have here. For instance, the profile nstcp_default_XA_XD_profile, which is intended for ICA-proxy traffic has some differences from the default profile, which are as follows: Window Scaling Selective Acknowledgement Forward Acknowledgement Use of Nagles Algorithm Window Scaling is a TCP option that allows the receiving point to accept more data than it is allowed in the TCP RFC for window size before getting an acknowledgement. By default, the window size is set to accept 65.536 bytes. With Window scaling enabled, it basically bitwise shifts the window size. This is an option that needs to be enabled on both endpoints in order to be used, and will only be sent in the initial three-way handshake. Select Acknowledgement (SACK) is a TCP option that allows for better handling of TCP retransmission. In a scenario of two hosts communicating where SACK is not enabled, and suddenly one of the hosts drops out of the network and loses some packets when they come back online it receives more packets from the other host. In this case the first host will ACK from the last packet it got from the other host before it dropped out. With SACK enabled, it will notify the other host of the last packet it got before it dropped out, and the other packets it received when he got back online. This allows for faster recovery of the communication, since the other host does not need to resend all the packets. Forward Acknowledgement (FACK) is a TCP option which works in conjunction with SACK and helps avoid TCP congestion by measuring the total number of data bytes that are outstanding in the network. Using the information from SACK, it can more precisely calculate how much data it can retransmit. Nagles Algoritm is a TCP feature that tries to cope with the small packet problem. Applications like Telnet often sends each keystroke within its own packet, creating multiple small packets containing only 1 byte of data, which results in a 41-byte packet for one keystroke. The algorithm works by combining a number of small outgoing messages into the same message, thus avoiding an overhead. Since ICA is a protocol that operates with many small packets, which might create congestion, Nagle is enabled in the TCP profile. Also, since many will be connecting using 3G or Wi-Fi, which might, in some cases, be unreliable to change channel, we need options that require the clients to be able to re-establish a connection quickly, which allows the use of SACK and FACK. Note that Nagle might have negative performance on applications that have their own buffering mechanism and operate inside LAN. If we take a look at another profile like nstcp_default_lan, we can see that FACK is disabled; this is because resources needed to calculate the amount of outstanding data in a high-speed network might be too much. Another important aspect of these profiles is the TCP congestion algorithms. For instance, nstcp_default_mobile uses the Westwood congestion algorithm; this is because it is much better at handling large bandwidth-delay paths, such as the wireless. The following congestion algorithms are available in NetScaler: Default (Based on TCP Reno) Westwood (Based on TCP Westwood+) BIC CUBIC Nile (Based on TCP Illinois) It is worth noting here that Westwood is aimed for 3G/4G connections or other slow wireless connections. BIC is aimed for high bandwidth connections with high latency, such as WAN connections. CUBIC is almost like BIC, but not as aggressive when it comes to fast-ramp and retransmissions. However, it is important to note that CUBIC is the default TCP algorithm in Linux kernels from 2.6.19 to 3.1 Nile is a newly created algorithm created by Citrix, which is based upon TCP Illinois and is targeted at high-speed, long-distance networks. It achieves higher throughput than standard TCP and is also compatible with standard TCP. So, here we can customize the algorithm that is better suited for a service. For instance, if we have a vServer that serves content to mobile devices, then we could use the nstcp_default_mobile TCP profile. There are also some other parameters that are important to keep in mind while working with the TCP profile. One of these parameters is multipath TCP. This is feature which allows endpoint which has multiple paths to a service. It is typically a mobile device which has WLAN and 3G capabilities, and allows the device to communicate with a service on a NetScaler using both channels at the same time. This requires that the device supports communication on both methods and that the service or application on the device supports Multipath TCP. So let's take an example of how a TCP profile might look like. If we have a vServer on NetScaler that is used to service an application to mobile devices. Meaning that the most common way that users can access this service is using 3G or Wi-Fi. The web service has its own buffering mechanism meaning that it tries not to send small packets over the link. The application is Multipath-TCP aware. In this scenario, we can leverage the nstcp_default_mobile profile, since it has most of the defaults for a mobile scenario, but we can also enable multipath TCP and create a new profile of it and bind it to the vServer. In order to bind a TCP profile to a vServer, we have go to a particular vServer | Edit | Profiles | TCP Profiles, as shown in the following screenshot: Note that AOL did a presentation of their own TCP customization on NetScaler; the presentation can be found at http://www.slideshare.net/masonke/net-scaler-tcpperformancetuningintheaolnetwork. It is also important to note that TCP should always be done in cooperation with the network team. Summary In this article, we have learned about Tuning for virtual environments and TCP tuning. Resources for Article: Further resources on this subject: Designing, Sizing, Building, and Configuring Citrix VDI-in-a-Box[article] XenMobile™ Solutions Bundle[article] Load balancing MSSQL [article]
Read more
  • 0
  • 0
  • 6089