How-To Tutorials

In this article by Rakesh Gupta, the author of the book Learning Salesforce Visual Workflow, we will discuss how to auto update child records using Process Builder of Salesforce. There are several business use cases where a customer wants to update child records based on some criteria, for example, auto-updating all related Opportunity to Closed-Lost if an account is updated to Inactive. To achieve these types of business requirements, you can use the Apex trigger. You can also achieve these types of requirements using the following methods: Process Builder A combination of Flow and Process Builder A combination of Flow and Inline Visualforce page on the account detail page (For more resources related to this topic, see here.) We will use Process Builder to solve these types of business requirements. Let's start with a business requirement. Here is a business scenario: Alice Atwood is working as a system administrator in Universal Container. She has received a requirement that once an account gets activated, the account phone must be synced with the related contact asst. phone field. This means whenever an account phone fields gets updated, the same phone number will be copied to the related contacts asst. phone field. Follow these instructions to achieve the preceding requirement using Process Builder: First of all, navigate to Setup | Build | Customize | Accounts | Fields and make sure that the Active picklist is available in your Salesforce organization. If it's not available, create a custom Picklist field with the name as Active, and enter the Yes and No values. To create a Process, navigate to Setup | Build | Create | Workflow & Approvals | Process Builder, click on New Button, and enter the following details: Name: Enter the name of the Process. Enter Update Contacts Asst Phone in Name. This must be within 255 characters. API Name: This will be autopopulated based on the name. Description: Write some meaningful text so that other developers or administrators can easily understand why this Process is created. The properties window will appear as shown in the following screenshot: Once you are done, click on the Save button. It will redirect you to the Process canvas, which allows you to create or modify the Process. After Define Process Properties, the next task is to select the object on which you want to create a Process and define the evaluation criteria. For this, click on the Add Object node. It will open an additional window on the right side of the Process canvas screen, where you have to enter the following details: Object: Start typing and then select the Account object. Start the process: For Start the process, select when a record is created or edited. This means the Process will fire every time, irrespective of record creation or updating. Allow process to evaluate a record multiple times in a single transaction?: Select this checkbox only when you want the Process to evaluate the same record up to five times in a single transaction. It might re-examine the record because a Process, Workflow Rule, or Flow may have updated the record in the same transaction. In this case, leave this unchecked. This window will appear as shown in the following screenshot: Once you are done with adding the Process criteria, click on the Save button. Similar to the Workflow Rule, once you save the panel, it doesn't allow you to change the selected object. After defining the evaluation criteria, the next step is to add the Process criteria. Once the Process criteria are true, only then will the Process execute the associated actions. To define the Process criteria, click on the Add Criteria node. It will open an additional window on the right side of the Process canvas screen, where you have to enter the following details: Criteria Name: Enter a name for the criteria node. Enter Update Contacts in Criteria Name. Criteria for Executing Actions: Select the type of criteria you want to define. You can use either a formula or a filter to define the Process criteria or no criteria. In this case, select Active equals to Yes. This means the Process will fire only when the account is active. This window will appear as shown in the following screenshot: Once you are done with defining the Process criteria, click on the Save button. Once you are done with the Process criteria node, the next step is to add an immediate action to update the related contact's asst. phone field. For this, we will use the Update Records action available under Process. Click on Add Action available under IMMEDIATE ACTIONS. It will open an additional window on the right side of the Process canvas screen, where you have to enter the following details: Action Type: Select the type of action. In this case, select Update Records. Action Name: Enter a name for this action. Enter Update Assts Phone in Action Name. Object: Start typing and then select the [Account].Contacts object. Field: Map the Asst. Phone field with the [Account]. Phone field. To select the fields, you can use field picker. To enter the value, use the text entry field. It will appear as shown in the following screenshot: Once you are done, click on the Save button. Once you are done with the immediate action, the final step is to activate it. To activate a Process, click on the Activate button available on the button bar. From now on, if you try to update an active account, Process will automatically update the related contact's asst. phone with the value available in the account phone field. Summary In this article, we have learned the technique of auto updating records in Process Builder. Resources for Article: Further resources on this subject: Visualforce Development with Apex [Article] Configuration in Salesforce CRM [Article] Introducing Salesforce Chatter [Article]

In this article we will cover etcd, CoreOS's central hub of all services that provides a reliable way of storing shared data across cluster machines and monitoring it. In this article, we will cover the following topics: Introducing etcd Reading and writing to etcd from the host machine Reading and writing from an application container Watching etcd changes A TTL (time to live) example use cases of etcd (For more resources related to this topic, see here.) Introducing etcd The etcd function is an open source distributed key value store on a computer network where information is stored on more than one node and data is replicated using the Raft consensus algorithm. The etcd function is used to store the CoreOS cluster service discovery and the shared configuration. The configuration is stored in the write-ahead log and includes the cluster member ID, cluster ID and cluster configuration, and everything else that is put there container applications running in the cluster. The etcd function runs on each cluster's central services role machine, and gracefully handles master election during network partitions and in the event of a loss of the current master. Reading and writing to etcd from the host machine You are going to learn how read and write to ectd from the host machine. We will use both the etcdctl and curl examples here. Logging in to host To login to CoreOS VM, follow these steps: Boot your CoreOS VM installed. In your terminal, type this: $ cdcoreos-vagrant $ vagrant up We need to login to the host via ssh: $ vagrant ssh Reading and writing to ectd Let's read and write to etcd using etcdctl. So, perform these steps: Set with etcdctl a message1 key with Book1 as the value: $ etcdctl set /message1 Book1Book1 (we got respond for our successful write to etcd Now, let's read the key value to double-check whether everything is fine there: $ etcdctl get /message1 Book1 Perfect! Next, let's try to do the same using curl via an HTTP-based API. The curl function is handy for accessing etcd from any place from where you have access to etcd cluster but don't want/need to use the etcdctl client: $ curl -L -X PUT http://127.0.0.1:2379/v2/keys/message2 -d value="Book2" {"action":"set","key":"/message2","prevValue":"Book1","value":"Book2","index":13371} Let's read it: $ curl -L http://127.0.0.1:2379/v2/keys/message2 {"action":"get","node":{"key":"/message2","value":"Book2","modifiedIndex":13371,"createdIndex":13371}} Using the HTTP-based etcd API means that etcd can be read from and written to by client applications without the need to interact with the command line. Now, if we want to delete the key value pair, we type the following command: $ etcdctl rm /message1 $ curl -L -X DELETE http://127.0.0.1:2379/v2/keys/message2 Also, we can add a key value pair to a directory, as directories are created automatically when a key is placed inside. We need only one command to put a key inside a directory: $ etcdctl set /foo-directory/foo-key somekey Let's now check the directory's content: $ etcdctl ls /foo-directory –recursive /foo-directory/foo-key Finally, we get the key value from the directory by typing: $ etcdctl get /foo-directory/foo-key somekey Reading and writing from the application container Usually, application containers (this is a general term for docker, rkt, and other types of containers) do not have etcdctl or even curl installed by default. Installing curl is much easier than installing etcdctl. For our example, we will use the AlpineLinux docker image, which is very small in size and will not take much time to pull from docker registry: Firstly, we need to check the docker0 interface IP, which we will use with curl: $ echo"$(ifconfig docker0 | awk'/<inet>/ { print $2}'):2379" 10.1.42.1:2379 Let's run the docker container with a bash shell: $ docker run -it alpine ash We should see something like this in Command Prompt:/ #. As curl is not installed by default on AlpineLinux, we need to install it: $ apk update&&apk add curl $ curl -L http://10.1.42.1:2379/v2/keys/ {"action":"get","node":{"key":"/","dir":true,"nodes":[{"key":"/coreos.com","dir":true,"modifiedIndex":3,"createdIndex":3}]}} Repeat steps 3 and 4 from the previous subtopic so that you understand that it does not matter where you are connecting to etcd from, curl still works in the same way. Press Ctrl +D to exit from the docker container. Watching changes in etcd This time, let's watch the key changes in etcd. Watching key changes is useful when we have, for example, one fleet unit with nginx writing its port to etcd, and another reverse proxy application watching for changes and updating its config: We need to create a directory in etcd first: $ etcdctlmkdir /foo-data Next, we watch for changes in this directory: $ etcdctl watch /foo-data--recursive Now open another CoreOS shell in a new terminal window: $ cdcoreos-vagrant $ vagrantssh We put a new key to /foo-data directory: $ etcdctl set /foo-data/Book is_cool In the first terminal, we should see a notification saying that the key was changed: is_cool A TTL (time to live) examples Sometimes, it is handy to put a time to live (TTL) for a key to expire in a certain amount of time. This is useful, for example,in the case of watching a key with a 60 second TTL, from a reverse proxy. So, if the nginx fleet service has not updated the key, it will expire in 60 seconds and will be removed from etcd. Then the reverse proxy checks for it and does not find it. Hence, it will remove the nginx service from config. Let's set TTL for 30 seconds in this example: Type this in a terminal: $ etcdctl set /foo "I'm Expiring in 30 sec" --ttl 30 I'm Expiring in 30 sec Verify that the key is still there: $ etcdctl get /foo I'm Expiring in 30 sec Check againafter 30 seconds : $ etcdctl get /foo If your requested key has already expired, you will be returned Error: 100: Error: 100: Key not found (/foo) [17053] This time the key got deleted by etcd because we put a TTL of 30 seconds for it. TTL is very handy to use to communicate between different services using etcd as the checking point. Use cases of etcd Application containers running on worker nodes with etcd in proxy mode can read and write to an etcd cluster. Very common etcd use cases are as follows: storing database connection settings, cache settings, and shared settings. For example, the Vulcand proxy server (http://vulcanproxy.com/) uses etcd to store web host connection details, and it becomes available for all cluster-connected worker machines. Another example could be to store a database password for MySQL and retrieve it when running an application container. Summary To learn more about CoreOS Essentials, the following books published by Packt Publishing (https://www.packtpub.com/) are recommended: Learning CoreOS (https://www.packtpub.com/networking-and-servers/learning-coreos) Resources for Article: Further resources on this subject: Mastering CentOS 7 Linux Server[article] Linux Shell Scripting[article] What is Kali Linux[article]

In this article by Alan Thorn author of the book Mastering Unity Scripting will cover the following topics: Events Event management (For more resources related to this topic, see here.) The Update events for MonoBehaviour objects seem to offer a convenient place for executing code that should perform regularly over time, spanning multiple frames, and possibly multiple scenes. When creating sustained behaviors over time, such as artificial intelligence for enemies or continuous motion, it may seem that there are almost no alternatives to filling an Update function with many if and switch statements, branching your code in different directions depending on what your objects need to do at the current time. But, when the Update events are seen this way, as a default place to implement prolonged behaviors, it can lead to severe performance problems for larger and more complex games. On deeper analysis, it's not difficult to see why this would be the case. Typically, games are full of so many behaviors, and there are so many things happening at once in any one scene that implementing them all through the Update functions is simply unfeasible. Consider the enemy characters alone, they need to know when the player enters and leaves their line of sight, when their health is low, when their ammo has expired, when they're standing on harmful terrain, when they're taking damage, when they're moving or not, and lots more. On thinking initially about this range of behaviors, it seems that all of them require constant and continuous attention because enemies should always know, instantly, when changes in these properties occur as a result of the player input. That is, perhaps, the main reason why the Update function seems to be the most suitable place in these situations but there are better alternatives, namely, event-driven programming. By seeing your game and your application in terms of events, you can make considerable savings in performance. This article then considers the issue of events and how to manage them game wide. Events Game worlds are fully deterministic systems; in Unity, the scene represents a shared 3D Cartesian space and timeline inside which finite GameObjects exist. Things only happen within this space when the game logic and code permits them to. For example, objects can only move when there is code somewhere that tells them to do so, and under specific conditions, such as when the player presses specific buttons on the keyboard. Notice from the example that behaviors are not simply random but are interconnected; objects move only when keyboard events occur. There is an important connection established between the actions, where one action entails another. These connections or linkages are referred to as events; each unique connection being a single event. Events are not active but passive; they represent moments of opportunity but not action in themselves, such as a key press, a mouse click, an object entering a collider volume, the player being attacked, and so on. These are examples of events and none of them say what the program should actually do, but only the kind of scenario that just happened. Event-driven programming starts with the recognition of events as a general concept and comes to see almost every circumstance in a game as an instantiation of an event; that is, as an event situated in time, not just an event concept but as a specific event that happens at a specific time. Understanding game events like these is helpful because all actions in a game can then be seen as direct responses to events as and when they happen. Specifically, events are connected to responses; an event happens and triggers a response. Further, the response can go on to become an event that triggers further responses and so on. In other words, the game world is a complete, integrated system of events and responses. Once the world is seen this way, the question then arises as to how it can help us improve performance over simply relying on the Update functions to move behaviors forward on every frame. And the method is simply by finding ways to reduce the frequency of events. Now, stated in this way, it may sound a crude strategy, but it's important. To illustrate, let's consider the example of an enemy character firing a weapon at the player during combat. Throughout the gameplay, the enemy will need to keep track of many properties. Firstly, their health, because when it runs low the enemy should seek out medical kits and aids to restore their health again. Secondly, their ammo, because when it runs low the enemy should seek to collect more and also the enemy will need to make reasoned judgments about when to fire at the player, such as only when they have a clear line of sight. Now, by simply thinking about this scenario, we've already identified some connections between actions that might be identified as events. But before taking this consideration further, let's see how we might implement this behavior using an Update function, as shown in the following code sample 4-1. Then, we'll look at how events can help us improve on that implementation: // Update is called once per frame void Update () {    //Check enemy health    //Are we dead?    if(Health <= 0)    {          //Then perform die behaviour          Die();          return;    }    //Check for health low    if(health <= 20)    {        //Health is low, so find first-aid          RunAndFindHealthRestore();          return;    }    //Check ammo    //Have we run out of ammo?    if(Ammo <= 0)    {          //Then find more          SearchMore();          return;    }    //Health and ammo are fine. Can we see player? If so, shoot    if(HaveLineOfSight)    {            FireAtPlayer();    } } The preceding code sample 4-1 shows a heavy Update function filled with lots of condition checking and responses. In essence, the Update function attempts to merge event handling and response behaviors into one and the results in an unnecessarily expensive process. If we think about the event connections between these different processes (the health and ammo check), we see how the code could be refactored more neatly. For example, ammo only changes on two occasions: when a weapon is fired or when new ammo is collected. Similarly, health only changes on two occasions: when an enemy is successfully attacked by the player or when an enemy collects a first-aid kit. In the first case, there is a reduction, and in the latter case, an increase. Since these are the only times when the properties change (the events), these are the only points where their values need to be validated. See the following code sample 4-2 for a refactored enemy, which includes C# properties and a much reduced Update function: using UnityEngine; using System.Collections; public class EnemyObject : MonoBehaviour {    //-------------------------------------------------------    //C# accessors for private variables    public int Health    {          get{return _health;}          set          {                //Clamp health between 0-100                _health = Mathf.Clamp(value, 0, 100);               //Check if dead                if(_health <= 0)                {                      OnDead();                      return;                }                //Check health and raise event if required                if(_health <= 20)               {                      OnHealthLow();                      return;                }          }    }    //-------------------------------------------------------    public int Ammo    {          get{return _ammo;}          set          {              //Clamp ammo between 0-50              _ammo = Mathf.Clamp(value,0,50);                //Check if ammo empty                if(_ammo <= 0)                {                      //Call expired event                      OnAmmoExpired();                      return;                }          }    }    //-------------------------------------------------------    //Internal variables for health and ammo    private int _health = 100;    private int _ammo = 50;    //-------------------------------------------------------    // Update is called once per frame    void Update ()    {    }    //-------------------------------------------------------    //This event is called when health is low    void OnHealthLow()    {          //Handle event response here    }    //-------------------------------------------------------    //This event is called when enemy is dead    void OnDead()    {        //Handle event response here    }    //-------------------------------------------------------    //Ammo run out event    void OnAmmoExpired()    {        //Handle event response here    }    //------------------------------------------------------- } The enemy class in the code sample 4-2 has been refactored to an event-driven design, where properties such as Ammo and Health are validated not inside the Update function but on assignment. From here, events are raised wherever appropriate based on the newly assigned values. By adopting an event-driven design, we introduce performance optimization and cleanness into our code; we reduce the excess baggage and value checks as found with the Update function in the code sample 4-1, and instead we only allow value-specific events to drive our code, knowing they'll be invoked only at the relevant times. Event management Event-driven programming can make our lives a lot easier. But no sooner than we accept events into the design do we come across a string of new problems that require a thoroughgoing resolution. Specifically, we saw in the code sample 4-2 how C# properties for health and ammo are used to validate and detect for relevant changes and then to raise events (such as OnDead) where appropriate. This works fine in principle, at least when the enemy must be notified about events that happen to itself. However, what if an enemy needed to know about the death of another enemy or needed to know when a specified number of other enemies had been killed? Now, of course, thinking about this specific case, we could go back to the enemy class in the code sample 4-2 and amend it to call an OnDead event not just for the current instance but for all other enemies using functions such as SendMessage. But this doesn't really solve our problem in the general sense. In fact, let's state the ideal case straight away; we want every object to optionally listen for every type of event and to be notified about them as and when they happen, just as easily as if the event had happened to them. So the question that we face now is about how to code an optimized system to allow easy event management like this. In short, we need an EventManager class that allows objects to listen to specific events. This system relies on three central concepts, as follows: Event Listener: A listener refers to any object that wants to be notified about an event when it happens, even its own events. In practice, almost every object will be a listener for at least one event. An enemy, for example, may want notifications about low health and low ammo among others. In this case, it's a listener for at least two separate events. Thus, whenever an object expects to be told when an event happens, it becomes a listener. Event Poster: In contrast to listeners, when an object detects that an event has occurred, it must announce or post a public notification about it that allows all other listeners to be notified. In the code sample 4-2, the enemy class detects the Ammo and Health events using properties and then calls the internal events, if required. But to be a true poster in this sense, we require that the object must raise events at a global level. Event Manager: Finally, there's an overarching singleton Event Manager object that persists across levels and is globally accessible. This object effectively links listeners to posters. It accepts notifications of events sent by posters and then immediately dispatches the notifications to all appropriate listeners in the form of events. Starting event management with interfaces The first or original entity in the event handling system is the listener—the thing that should be notified about specific events as and when they happen. Potentially, a listener could be any kind of object or any kind of class; it simply expects to be notified about specific events. In short, the listener will need to register itself with the Event Manager as a listener for one or more specific events. Then, when the event actually occurs, the listener should be notified directly by a function call. So, technically, the listener raises a type-specificity issue for the Event Manager about how the manager should invoke an event on the listener if the listener could potentially be an object of any type. Of course, this issue can be worked around, as we've seen, using either SendMessage or BroadcastMessage. Indeed, there are event handling systems freely available online, such as NotificationCenter that rely on these functions. However, we'll avoid them using interfaces and use polymorphism instead, as both SendMessage and BroadcastMessage rely heavily on reflection. Specifically, we'll create an interface from which all listener objects derive. More information on the freely available NotificationCenter (C# version) is available from the Unity wiki at http://wiki.unity3d.com/index.php?title=CSharpNotificationCenter. In C#, an interface is like a hollow abstract base class. Like a class, an interface brings together a collection of methods and functions into a single template-like unit. But, unlike a class, an interface only allows you to define function prototypes such as the name, return type, and arguments for a function. It doesn't let you define a function body. The reason being that an interface simply defines the total set of functions that a derived class will have. The derived class may implement the functions however necessary, and the interface simply exists so that other objects can invoke the functions via polymorphism without knowing the specific type of each derived class. This makes interfaces a suitable candidate to create a Listener object. By defining a Listener interface from which all objects will be derived, every object has the ability to be a listener for events. The following code sample 4-3 demonstrates a sample Listener interface: 01 using UnityEngine; 02 using System.Collections; 03 //----------------------------------------------------------- 04 //Enum defining all possible game events 05 //More events should be added to the list 06 public enum EVENT_TYPE {GAME_INIT, 07                                GAME_END, 08                                 AMMO_EMPTY, 09                                 HEALTH_CHANGE, 10                                 DEAD}; 11 //----------------------------------------------------------- 12 //Listener interface to be implemented on Listener classes 13 public interface IListener 14 { 15 //Notification function invoked when events happen 16 void OnEvent(EVENT_TYPE Event_Type, Component Sender,    Object Param = null); 17 } 18 //----------------------------------------------------------- The following are the comments for the code sample 4-3: Lines 06-10: This enumeration should define a complete list of all possible game events that could be raised. The sample code lists only five game events: GAME_INIT, GAME_END, AMMO_EMPTY, HEALTH_CHANGE, and DEAD. Your game will presumably have many more. You don't actually need to use enumerations for encoding events; you could just use integers. But I've used enumerations to improve event readability in code. Lines 13-17: The Listener interface is defined as IListener using the C# interfaces. It supports just one event, namely OnEvent. This function will be inherited by all derived classes and will be invoked by the manager whenever an event occurs for which the listener is registered. Notice that OnEvent is simply a function prototype; it has no body. More information on C# interfaces can be found at http://msdn.microsoft.com/en-us/library/ms173156.aspx. Using the IListener interface, we now have the ability to make a listener from any object using only class inheritance; that is, any object can now declare itself as a listener and potentially receive events. For example, a new MonoBehaviour component can be turned into a listener with the following code sample 4-4. This code uses multiple inheritance, that is, it inherits from two classes. More information on multiple inheritance can be found at http://www.dotnetfunda.com/articles/show/1185/multiple-inheritance-in-csharp: using UnityEngine; using System.Collections; public class MyCustomListener : MonoBehaviour, IListener {    // Use this for initialization    void Start () {}    // Update is called once per frame    void Update () {}    //---------------------------------------    //Implement OnEvent function to receive Events    public void OnEvent(EVENT_TYPE Event_Type, Component Sender, Object Param = null)    {    }    //--------------------------------------- } Creating an EventManager Any object can now be turned into a listener, as we've seen. But still the listeners must register themselves with a manager object of some kind. Thus, it is the duty of the manager to call the events on the listeners when the events actually happen. Let's now turn to the manager itself and its implementation details. The manager class will be called EventManager, as shown in the following code sample 4-5. This class, being a persistent singleton object, should be attached to an empty GameObject in the scene where it will be directly accessible to every other object through a static instance property. More on this class and its usage is considered in the subsequent comments: 001 using UnityEngine; 002 using System.Collections; 003 using System.Collections.Generic; 004 //----------------------------------- 005 //Singleton EventManager to send events to listeners 006 //Works with IListener implementations 007 public class EventManager : MonoBehaviour 008 { 009     #region C# properties 010 //----------------------------------- 011     //Public access to instance 012     public static EventManager Instance 013       { 014             get{return instance;} 015            set{} 016       } 017   #endregion 018 019   #region variables 020       // Notifications Manager instance (singleton design pattern) 021   private static EventManager instance = null; 022 023     //Array of listeners (all objects registered for events) 024     private Dictionary<EVENT_TYPE, List<IListener>> Listeners          = new Dictionary<EVENT_TYPE, List<IListener>>(); 025     #endregion 026 //----------------------------------------------------------- 027     #region methods 028     //Called at start-up to initialize 029     void Awake() 030     { 031             //If no instance exists, then assign this instance 032             if(instance == null) 033           { 034                   instance = this; 035                   DontDestroyOnLoad(gameObject); 036           } 037             else 038                   DestroyImmediate(this); 039     } 040//----------------------------------------------------------- 041     /// <summary> 042     /// Function to add listener to array of listeners 043     /// </summary> 044     /// <param name="Event_Type">Event to Listen for</param> 045     /// <param name="Listener">Object to listen for event</param> 046     public void AddListener(EVENT_TYPE Event_Type, IListener        Listener) 047    { 048           //List of listeners for this event 049           List<IListener> ListenList = null; 050 051           // Check existing event type key. If exists, add to list 052           if(Listeners.TryGetValue(Event_Type,                out ListenList)) 053           { 054                   //List exists, so add new item 055                   ListenList.Add(Listener); 056                   return; 057           } 058 059           //Otherwise create new list as dictionary key 060           ListenList = new List<IListener>(); 061           ListenList.Add(Listener); 062           Listeners.Add(Event_Type, ListenList); 063     } 064 //----------------------------------------------------------- 065       /// <summary> 066       /// Function to post event to listeners 067       /// </summary> 068       /// <param name="Event_Type">Event to invoke</param> 069       /// <param name="Sender">Object invoking event</param> 070       /// <param name="Param">Optional argument</param> 071       public void PostNotification(EVENT_TYPE Event_Type,          Component Sender, Object Param = null) 072       { 073           //Notify all listeners of an event 074 075           //List of listeners for this event only 076           List<IListener> ListenList = null; 077 078           //If no event exists, then exit 079           if(!Listeners.TryGetValue(Event_Type,                out ListenList)) 080                   return; 081 082             //Entry exists. Now notify appropriate listeners 083             for(int i=0; i<ListenList.Count; i++) 084             { 085                   if(!ListenList[i].Equals(null)) 086                   ListenList[i].OnEvent(Event_Type, Sender, Param); 087             } 088     } 089 //----------------------------------------------------------- 090     //Remove event from dictionary, including all listeners 091     public void RemoveEvent(EVENT_TYPE Event_Type) 092     { 093           //Remove entry from dictionary 094           Listeners.Remove(Event_Type); 095     } 096 //----------------------------------------------------------- 097       //Remove all redundant entries from the Dictionary 098     public void RemoveRedundancies() 099     { 100             //Create new dictionary 101             Dictionary<EVENT_TYPE, List<IListener>>                TmpListeners = new Dictionary                <EVENT_TYPE, List<IListener>>(); 102 103             //Cycle through all dictionary entries 104             foreach(KeyValuePair<EVENT_TYPE, List<IListener>>                Item in Listeners) 105             { 106                   //Cycle all listeners, remove null objects 107                   for(int i = Item.Value.Count-1; i>=0; i--) 108                   { 109                         //If null, then remove item 110                         if(Item.Value[i].Equals(null)) 111                                 Item.Value.RemoveAt(i); 112                   } 113 114           //If items remain in list, then add to tmp dictionary 115                   if(Item.Value.Count > 0) 116                         TmpListeners.Add (Item.Key,                              Item.Value); 117             } 118 119             //Replace listeners object with new dictionary 120             Listeners = TmpListeners; 121     } 122 //----------------------------------------------------------- 123       //Called on scene change. Clean up dictionary 124       void OnLevelWasLoaded() 125       { 126           RemoveRedundancies(); 127       } 128 //----------------------------------------------------------- 129     #endregion 130 } More information on the OnLevelWasLoaded event can be found at http://docs.unity3d.com/ScriptReference/MonoBehaviour.OnLevelWasLoaded.html. The following are the comments for the code sample 4-5: Line 003: Notice the addition of the System.Collections.Generic namespace giving us access to additional mono classes, including the Dictionary class. This class will be used throughout the EventManager class. In short, the Dictionary class is a special kind of 2D array that allows us to store a database of values based on key-value pairing. More information on the Dictionary class can be found at http://msdn.microsoft.com/en-us/library/xfhwa508%28v=vs.110%29.aspx. Line 007: The EventManager class is derived from MonoBehaviour and should be attached to an empty GameObject in the scene where it will exist as a persistent singleton. Line 024: A private member variable Listeners is declared using a Dictionary class. This structure maintains a hash-table array of key-value pairs, which can be looked up and searched like a database. The key-value pairing for the EventManager class takes the form of EVENT_TYPE and List<Component>. In short, this means that a list of event types can be stored (such as HEALTH_CHANGE), and for each type there could be none, one, or more components that are listening and which should be notified when the event occurs. In effect, the Listeners member is the primary data structure on which the EventManager relies to maintain who is listening for what. Lines 029-039: The Awake function is responsible for the singleton functionality, that is, to make the EventManager class into a singleton object that persists across scenes. Lines 046-063: The AddListener method of EventManager should be called by a Listener object once for each event for which it should listen. The method accepts two arguments: the event to listen for (Event_Type) and a reference to the listener object itself (derived from IListener), which should be notified if and when the event happens. The AddListener function is responsible for accessing the Listeners dictionary and generating a new key-value pair to store the connection between the event and the listener. Lines 071-088: The PostNotification function can be called by any object, whether a listener or not, whenever an event is detected. When called, the EventManager cycles all matching entries in the dictionary, searching for all listeners connected to the current event, and notifies them by invoking the OnEvent method through the IListener interface. Lines 098-127: The final methods for the EventManager class are responsible for maintaining data integrity of the Listeners structure when a scene change occurs and the EventManager class persists. Although the EventManager class persists across scenes, the listener objects themselves in the Listeners variable may not do so. They may get destroyed on scene changes. If so, scene changes will invalidate some listeners, leaving the EventManager with invalid entries. Thus, the RemoveRedundancies method is called to find and eliminate all invalid entries. The OnLevelWasLoaded event is invoked automatically by Unity whenever a scene change occurs. More information on the OnLevelWasLoaded event can be found online at: http://docs.unity3d.com/ScriptReference/MonoBehaviour.OnLevelWasLoaded.html. #region and #endregion The two preprocessor directives #region and #endregion (in combination with the code folding feature) can be highly useful for improving the readability of your code and also for improving the speed with which you can navigate the source file. They add organization and structure to your source code without affecting its validity or execution. Effectively, #region marks the top of a code block and #endregion marks the end. Once a region is marked, it becomes foldable, that is, it becomes collapsible using the MonoDevelop code editor, provided the code folding feature is enabled. Collapsing a region of code is useful for hiding it from view, which allows you to concentrate on reading other areas relevant to your needs, as shown in the following screenshot: Enabling code folding in MonoDevelop To enable code folding in MonoDevelop, select Options in Tools from the application menu. This displays the Options window. From here, choose the General tab in the Text Editor option and click on Enable code folding as well as Fold #regions by default. Using EventManager Now, let's see how to put the EventManager class to work in a practical context from the perspective of listeners and posters in a single scene. First, to listen for an event (any event) a listener must register itself with the EventManager singleton instance. Typically, this will happen once and at the earliest opportunity, such as the Start function. Do not use the Awake function; this is reserved for an object's internal initialization as opposed to the functionality that reaches out beyond the current object to the states and setup of others. See the following code sample 4-6 and notice that it relies on the Instance static property to retrieve a reference to the active EventManager singleton: //Called at start-up void Start() { //Add myself as listener for health change events EventManager.Instance.AddListener(EVENT_TYPE.HEALTH_CHANGE, this); } Having registered listeners for one or more events, objects can then post notifications to EventManager as events are detected, as shown in the following code sample 4-7: public int Health { get{return _health;} set {    //Clamp health between 0-100    _health = Mathf.Clamp(value, 0, 100);    //Post notification - health has been changed   EventManager.Instance. PostNotification(EVENT_TYPE.HEALTH_CHANGE, this, _health); } } Finally, after a notification is posted for an event, all the associated listeners are updated automatically through EventManager. Specifically, EventManager will call the OnEvent function of each listener, giving listeners the opportunity to parse event data and respond where needed, as shown in the following code sample 4-7: //Called when events happen public void OnEvent(EVENT_TYPE Event_Type, Component Sender, object Param = null) { //Detect event type switch(Event_Type) {    case EVENT_TYPE.HEALTH_CHANGE:          OnHealthChange(Sender, (int)Param);    break; } } Summary This article focused on the manifold benefits available for your applications by adopting an event-driven framework consistently through the EventManager class. In implementing such a manager, we were able to rely on either interfaces or delegates, and either method is powerful and extensible. Specifically, we saw how it's easy to add more and more functionality into an Update function but how doing this can lead to severe performance issues. Better is to analyze the connections between your functionality to refactor it into an event-driven framework. Essentially, events are the raw material of event-driven systems. They represent a necessary connection between one action (the cause) and another (the response). To manage events, we created the EventManager class—an integrated class or system that links posters to listeners. It receives notifications from posters about events as and when they happen and then immediately dispatches a function call to all listeners for the event. Resources for Article: Further resources on this subject: Customizing skin with GUISkin [Article] 2D Twin-stick Shooter [Article] Components in Unity [Article]

Business intelligence is big business. Salesforce’s purchase of Tableau earlier this year (for a cool $16 billion) proves the value of a powerful data analytics platform, and demonstrates how the business intelligence space is reshaping expectations and demands in the established CRM and ERP marketplace. To a certain extent, the amount Salesforce paid for Tableau highlights that when it comes to business intelligence, tooling is paramount. Without a tool that fits the needs and skill levels of those that need BI and use analytics, discussions around architecture and strategy are practically moot. So, what are the best business intelligence tools? And more importantly, how do they differ from one another? Which one is right for you? Read next: 4 important business intelligence considerations for the rest of 2019 The best business intelligence tools 2019 Tableau Let’s start with the obvious one: Tableau. It’s one of the most popular business intelligence tools on the planet, and with good reason; it makes data visualization and compelling data storytelling surprisingly easy. With a drag and drop interface, Tableau requires no coding knowledge from users. It also allows users to ask ‘what if’ scenarios to model variable changes, which means you can get some fairly sophisticated insights with just a few simple interactions. But while Tableau is undoubtedly designed to be simple, it doesn’t sacrifice complexity. Unlike other business intelligence tools, Tableau allows users to include an unlimited number of datapoints in their analytics projects. When should you use Tableau and how much does it cost? The core Tableau product is aimed at data scientists and data analysts who want to be able to build end-to-end analytics pipelines. You can trial the product for free for 14 days, but this will then cost you $70/month. This is perhaps one of the clearest use cases - if you’re interested and passionate about data, Tableau practically feels like a toy. However, for those that want to employ Tableau across their organization, the product offers a neat pricing tier. Tableau Creator is built for individual power users - like those described above, Tableau Explorer for self-service analytics, and Tableau Viewer for those that need access to Tableau for limited access to dashboard and analytics. Tableau eBooks and videos Mastering Tableau 2019.1 - Second Edition Tableau 2019.x Cookbook Getting Started with Tableau 2019.2 - Second Edition Tableau in 7 Steps [Video] PowerBI PowerBI is Microsoft’s business intelligence platform. Compared to Tableau it is designed more for reporting and dashboards rather than data exploration and storytelling. If you use a wide range of Microsoft products, PowerBI is particularly powerful. It can become a centralized space for business reporting and insights. Like Tableau, it’s also relatively easy to use. With support from Microsoft Cortana - the company’s digital assistant - it’s possible to perform natural language queries. When should you use PowerBI and how much does it cost? PowerBI is an impressive business intelligence product. But to get the most value, you need to be committed to Microsoft. This isn’t to say you shouldn’t be - the company has been on form in recent years and appears to really understand what modern businesses and users need. On a similar note, a good reason to use PowerBI is for unified and aligned business insights. If Tableau is more suited to personal exploration, or project-based storytelling, PowerBI is an effective option for organizations that want more clarity and shared visibility on key performance metrics. This is reflected in the price. For personal users the desktop version of PowerBI is free, while a pro license is $9.99 a month. A premium plan which includes cloud resources (storage and compute) starts at $4,995. This is the option for larger organizations that are fully committed to the Microsoft suite and has a clear vision of how it wants to coordinate analytics and reporting. PowerBI eBooks and videos Learn Power BI Microsoft Power BI Quick Start Guide Learning Microsoft Power BI [Video] Qlik Sense and QlikView Okay, so here we’re going to include two business intelligence products together: Qlik Sense and QlikView. Obviously, they’re both part of the same family - they’re built by business intelligence company Qlik. More importantly, they’re both quite different products. What’s the difference between Qlik Sense and QlikView? As we’ve said, Qlik Sense and QlikView are two different products. QlikView is the older and more established tool. It’s what’s usually described as a ‘guided analytics’ platform, which means dashboards and analytics applications can be built for end users. The tool gives freedom to engineers and data scientists to build what they want but doesn’t allow end users to ‘explore’ data in any more detail than what is provided. QlikView is quite a sophisticated platform and is widely regarded as being more complex to use than Tableau or PowerBI. While PowerBI or Tableau can be used by anyone with an intermediate level of data literacy and a willingness to learn, QlikView will always be the preserve of data scientists and analysts. This doesn’t make it a poor choice. If you know how to use it properly, QlikView can provide you with more in-depth analysis than any other business intelligence platforms, helping users to see patterns and relationships across different data sets. If you’re working with big data, for example, and you have a team of data scientists and data engineers, QlikView is a good option. Qlik Sense, meanwhile, could be seen as Qlik’s attempt to compete with the likes of Tableau and PowerBI. It’s a self-service BI tool, which allows end users to create their own data visualisations and explore data through a process of ‘data discovery’. When should you use QlikView and how much does it cost? QlikView should be used when you need to build a cohesive reporting and business intelligence solution. It’s perfect for when you need a space to manage KPIs and metrics across different teams. Although a free edition is available for personal use, Qlik doesn’t publish prices for enterprise users. You’ll need to get in touch with the company’s sales team to purchase. QlikView eBooks and videos QlikView: Advanced Data Visualization QlikView Dashboard Development [Video] When should you use Qlik Sense and how much does it cost? Qlik Sense should be used when you have an organization full of people curious and prepared to get their hands on their data. If you already have an established model of reporting performance, Qlik Sense is a useful extra that can give employees more autonomy over how data can be used. When it comes to pricing, Qlik Sense is one of the more complicated business intelligence options. Like QlikView, there’s a free option for personal use, and again like QlikView, there’s no public price available - so you’ll have to connect with Qlik directly. To add an additional layer of complexity, there’s also a product called ‘Cloud Basic’ - this is free and can be shared between up to 5 users. It’s essentially a SaaS version of the Qlik Sense product. If you need to add more than 5 users, it costs $15 per user/month. Qlik Sense eBooks and videos Mastering Qlik Sense [Video] Qlik Sense Cookbook - Second Edition Data Storytelling with Qlik Sense [Video] Hands-On Business Intelligence with Qlik Sense Read next: Top 5 free Business Intelligence tools Splunk Splunk isn’t just a business intelligence tool. To a certain extent, it’s related to application monitoring and logging tools such as Datadog, New Relic, and AppDynamics. It’s built for big data and real-time analytics, which means that it’s well-suited to offering insights on business processes and product performance. The copy on the Splunk website talks about “real-time visibility across the enterprise” and describes Splunk as a “data-to-everything” platform. The product, then, is pitching itself as something that can embed itself inside existing systems, and bring insight and intelligence to places and spaces where it’s particularly valuable. This is in contrast to PowerBI and Tableau, which are designed for exploration and accessibility. This isn’t to say that Splunk doesn’t enable sophisticated data exploration, but rather that it is geared towards monitoring systems and processes, understanding change. It’s a tool built for companies that need full transparency - or, in other words, dynamic operational intelligence. When should you use Splunk and how much does it cost? Splunk is a tool that should be used if you’re dealing with dynamic and real-time data. If you want to be able to model and explore wide-ranging existing sets of data Tableau or PowerBI are probably a better bet. But if you need to be able to make decisions in an active and ongoing scenario, Splunk is a tool that can provide substantial support. The reason that Splunk is included as a part of this list of business intelligence tools is because real-time visibility and insight is vital for businesses. Typically understanding application performance or process efficiency might have been embedded within particular departments, such as a centralized IT function. Now, with businesses dependent upon operational excellence, and security and reliability in the digital arena becoming business critical, Splunk is a tool that deserves its status inside (and across) organizations. Splunk’s pricing is complicated. Prices are generally dependent on how much data you want to index - or, in other words, how much you’re giving Splunk to deal with. But to add to that, Splunk also have a perpetual license ( a one time payment) and an annual term license, which needs to be renewed. So, you can index 1GB/day for $4,500 on a perpetual license, or $1,800 on an annual license. If you want to learn more about Splunk’s pricing option, this post is very useful. Splunk eBooks and videos Splunk 7 Essentials [E-Learning] Splunk 7.x Quick Start Guide Splunk Operational Intelligence Cookbook - Third Edition IBM Cognos IBM Cognos is IBM’s flagship business intelligence tool. It’s probably best viewed as existing somewhere between PowerBI and Tableau. It’s designed for reporting dashboards that allow monitoring and analytics, but it is nevertheless also intended for self-service. To that end, you might say it’s more limited in capabilities than PowerBI, but it’s nevertheless more accessible for non-technical end users to explore data. It’s also relatively easy to integrate with other systems and data sources. So, if your data is stored in Microsoft or Oracle cloud services and databases, it’s relatively straightforward to get started with IBM Cognos. However, it’s worth noting that despite the accesibility of IBM’s product, it still needs centralized control and implementation. It doesn’t offer the level of ease that you get with Tableau, for example. When should you use IBM Cognos and how much does it cost? Cognos is perhaps the go-to option if PowerBI and Tableau don’t quite work for you. Perhaps you like the idea of Tableau but need more centralization. Or maybe you need a strong and cohesive reporting system but don’t feel prepared to buy into Microsoft. This isn’t to make IBM Cognos sound like the outsider - in fact, from an efficiency perspective it’s possibly the best way to ensure to ensure some degree of portability between data sources and to manage the age-old problem of data silos. If you’re not quite sure what business intelligence tool is right for you, it’s well worth taking advantage of Cognos’s free trial - you get unlimited access for a month. If you like what you get, you then have a choice between a premium version - which costs $70 per user/month, and the enterprise plan, the price of which isn’t publicly available. IBM Cognos eBooks and videos IBM Cognos Framework Manager [Video] IBM Cognos Report Studio [Video] IBM Cognos Connection and Workspace Advanced [Video] Conclusion: To choose the best business intelligence solution for your organization, you need to understand your needs and goals Business intelligence is a crowded market. The products listed here are the tip of the iceberg when it comes to analytics, monitoring, and data visualization. This is good and bad - it means there are plenty of options and opportunities, but it also means that sorting through the options to find the right one might take up some of your time. That’s okay though - if possible, try to take advantage of free trial periods. And if you’re in a rush to get work done, use them on active projects. You could even allocate different platforms and tools to different team members and get them to report on what worked well and what didn’t. That way you can have documented insights on how the products might actually be used within the organization. This will help you to better reach a conclusion about the best tool for the job. Business intelligence done well can be extremely valuable - so don’t waste money and don’t waste time on tools that aren’t going to deliver what you need.

(For more resources related to this topic, see here.) The latest version of the example code for this article can be found at http://github.com/CamelCookbook/camel-cookbook-examples. You can also download the example code files for all Packt books you have purchased from your account at https://www.packtpub.com. If you purchased this book elsewhere, you can visit https://www.packtpub.com/books/content/support and register to have the files e-mailed directly to you. In this article we will explore a number of ways in which Camel performs message content transformation: Let us first look at some important concepts regarding transformation of messages in Camel: Using the transform statement. This allows you to reference Camel Expression Language code within the route to do message transformations. Calling a templating component, such as Camel's XSLT or Velocity template style components. This will typically reference an external template resource that is used in transforming your message. Calling a Java method (for example, beanref), defined by you, within a Camel route to perform the transformation. This is a special case processor that can invoke any referenced Java object method. Camel's Type Converter capability that can automatically cast data from one type to another transparently within your Camel route. This capability is extensible, so you can add your own Type Converters. Camel's Data Format capability that allows us to use built-in, or add our own, higher order message format converters. A Camel Data Format goes beyond simple data type converters, which handle simple data type translations such as String to int, or File to String. Data Formats are used to translate between a low-level representation (XML) and a high-level one (Java objects). Other examples include encrypting/decrypting data, and compressing/decompressing data. For more, see http://camel.apache.org/data-format.html. A number of Camel architectural concepts are used throughout this article. Full details can be found at the Apache Camel website at http://camel.apache.org. The code for this article is contained within the camel-cookbook-transformation module of the examples. Transforming using a Simple Expression When you want to transform a message in a relatively straightforward way, you use Camel's transform statement along with one of the Expression Languages provided by the framework. For example, Camel's Simple Expression Language provides you with a quick, inline mechanism for straightforward transformations. This recipe will show you how to use Camel's Simple Expression Language to transform the message body. Getting ready The Java code for this recipe is located in the org.camelcookbook.transformation.simple package. Spring XML files are located under src/main/resources/META-INF/spring and are prefixed with simple. How to do it... In a Camel route, use a transform DSL statement containing the Expression Language code to do your transformation. In the XML DSL, this is written as follows: <route> <from uri="direct:start"/> <transform> <simple>Hello ${body}</simple> </transform> </route> In the Java DSL, the same route is expressed as: from("direct:start") .transform(simple("Hello ${body}")); In this example, the message transformation prefixes the incoming message with the phrase Hello using the Simple Expression Language. The processing step after the transform statement will see the transformed message content in the body of the exchange. How it works... Camel's Simple Expression Language is quite good at manipulating the String content through its access to all aspects of the message being processed, through its rich String and logical operators. The result of your Simple Expression becomes the new message body after the transform step. This includes predicates such as using Simple's logical operators, to evaluate a true or false condition; the results of that Boolean operation become the new message body containing a String: "true" or "false". The advantage of using a distinct transform step within a route, as opposed to embedding it within a processor, is that the logic is clearly visible to other programmers. Ensure that the expression embedded within your route is kept simple so as to not distract the next developer from the overall purpose of the integration. It is best to move more complex (or just lengthy) transformation logic into its own subroute, and invoke it using direct: or seda:. There's more... The transform statement will work with any Expression Language available in Camel, so if you need more powerful message processing capabilities you can leverage scripting languages such as Groovy or JavaScript (among many others) as well. The Transforming inline with XQuery recipe will show you how to use the XQuery Expression Language to do transformations on XML messages. See also Message Translator: http://camel.apache.org/message-translator.html Camel Expression capabilities: http://camel.apache.org/expression.html Camel Simple Expression Language: http://camel.apache.org/simple.html Languages supported by Camel: http://camel.apache.org/languages.html The Transforming inline with XQuery recipe   Transforming inline with XQuery Camel supports the use of Camel's XQuery Expression Language along with the transform statement as a quick and easy way to transform an XML message within a route. This recipe will show you how to use an XQuery Expression to do in-route XML transformation. Getting ready The Java code for this recipe is located in the org.camelcookbook.transformation.xquery package. Spring XML files are located under src/main/resources/META-INF/spring and prefixed with xquery. To use the XQuery Expression Language, you need to add a dependency element for the camel-saxon library, which provides the implementation for the XQuery Expression Language. Add the following to the dependencies section of your Maven POM: <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-saxon</artifactId> <version>${camel-version}</version> </dependency>   How to do it... In the Camel route, specify a transform statement followed by the XQuery Expression Language code to do your transformation. In the XML DSL, this is written as: <route> <from uri="direct:start"/> <transform> <xquery> <books>{ for $x in /bookstore/book where $x/price>30 order by $x/title return $x/title }</books> </xquery> </transform> </route> When using the XML DSL, remember to XML encode the XQuery embedded XML elements. Therefore, < becomes &lt; and > becomes &gt;. In the Java DSL, the same route is expressed as: from("direct:start") .transform(xquery("<books>{ for $x in /bookstore/book " + "where $x/price>30 order by $x/title " + "return $x/title }</books>")); Feed the following input XML message through the transformation: <bookstore> <book category="COOKING"> <title lang="en">Everyday Italian</title> <author>Giada De Laurentiis</author> <year>2005</year> <price>30.00</price> </book> <book category="CHILDREN"> <title lang="en">Harry Potter</title> <author>J K. Rowling</author> <year>2005</year> <price>29.99</price> </book> <book category="PROGRAMMING"> <title lang="en">Apache Camel Developer's Cookbook</title> <author>Scott Cranton</author> <author>Jakub Korab</author> <year>2013</year> <price>49.99</price> </book> <book category="WEB"> <title lang="en">Learning XML</title> <author>Erik T. Ray</author> <year>2003</year> <price>39.95</price> </book> </bookstore> The resulting message will be: <books> <title lang="en">Apache Camel Developer's Cookbook</title> <title lang="en">Learning XML</title> </books> The processing step after the transform statement will see the transformed message content in the body of the exchange. How it works... Camel's XQuery Expression Language is a good way to inline XML transformation code within your route. The result of the XQuery Expression becomes the new message body after the transform step. All of the message's body, headers, and properties are made available to the XQuery Processor, so you can reference them directly within your XQuery statement. This provides you with a powerful mechanism for transforming XML messages. If you are more comfortable with XSLT, take a look at the Transforming with XSLT recipe. In-lining the transformation within your integration route can sometimes be an advantage as you can clearly see what is being changed. However, when the transformation expression becomes so complex that it starts to overwhelm the integration route, you may want to consider moving the transformation expression outside of the route. See the Transforming using a Simple Expression recipe for another inline transformation example, and see the Transforming with XSLT recipe for an example of externalizing your transformation. You can fetch the XQuery Expression from an external file using Camel's resource reference syntax. To reference an XQuery file on the classpath you can specify: <transform> <xquery>resource:classpath:/path/to/myxquery.xml</xquery> </transform> This is equivalent to using XQuery as an endpoint: <to uri="xquery:classpath:/path/to/myxquery.xml"/>   There's more... The XQuery Expression Language allows you to pass in headers associated with the message. These will show up as XQuery variables that can be referenced within your XQuery statements. Consider, from the previous example, to allow the value of the books that are filtered to be passed in with the message body, that is, parameterize the XQuery, you can modify the XQuery statement as follows: <transform> <xquery> declare variable $in.headers.myParamValue as xs:integer external; <books value='{$in.headers.myParamValue}'&gt;{ for $x in /bookstore/book where $x/price>$in.headers.myParamValue order by $x/title return $x/title }&lt;/books&gt; </xquery> </transform> Message headers will be associated with an XQuery variable called in.headers.<name of header>. To use this in your XQuery, you need to explicitly declare an external variable of the same name and XML Schema (xs:) type as the value of the message header. The transform statement will work with any Expression Language enabled within Camel, so if you need more powerful message processing capabilities you can leverage scripting languages such as Groovy or JavaScript (among many others) as well. The Transforming using a Simple Expression recipe will show you how to use the Simple Expression Language to do transformations on String messages. See also Message Translator: http://camel.apache.org/message-translator.html Camel Expression capabilities: http://camel.apache.org/expression.html Camel XQuery Expression Language: http://camel.apache.org/xquery.html XQuery language: http://www.w3.org/XML/Query/ Languages supported by Camel: http://camel.apache.org/languages.html The Transforming with XSLT recipe The Transforming using a Simple Expression recipe   Transforming with XSLT When you want to transform an XML message using XSLT, use Camel's XSLT Component. This is similar to the Transforming inline with XQuery recipe except that there is no XSLT Expression Language, so it can only be used as an endpoint. This recipe will show you how to transform a message using an external XSLT resource. Getting ready The Java code for this recipe is located in the org.camelcookbook.transformation.xslt package. Spring XML files are located under src/main/resources/META-INF/spring and prefixed with xslt. How to do it... In a Camel route, add the xslt processor step into the route at the point where you want the XSLT transformation to occur. The XSLT file must be referenced as an external resource, and depending on where the file is located, prefixed with either classpath:(default if not using a prefix), file:, or http:. In the XML DSL, this is written as: <route> <from uri="direct:start"/> <to uri="xslt:book.xslt"/> </route> In the Java DSL, the same route is expressed as: from("direct:start") .to("xslt:book.xslt"); The next processing step in the route will see the transformed message content in the body of the exchange. How it works... The following example shows how the preceding steps will process an XML file. Consider the following input XML message: <bookstore> <book category="COOKING"> <title lang="en">Everyday Italian</title> <author>Giada De Laurentiis</author> <year>2005</year> <price>30.00</price> </book> <book category="CHILDREN"> <title lang="en">Harry Potter</title> <author>J K. Rowling</author> <year>2005</year> <price>29.99</price> </book> <book category="PROGRAMMING"> <title lang="en">Apache Camel Developer's Cookbook</title> <author>Scott Cranton</author> <author>Jakub Korab</author> <year>2013</year> <price>49.99</price> </book> <book category="WEB"> <title lang="en">Learning XML</title> <author>Erik T. Ray</author> <year>2003</year> <price>39.95</price> </book> </bookstore> Process this with the following XSLT contained in books.xslt: <?xml version="1.0" encoding="UTF-8"?> <xsl:stylesheet version="1.0" > <xsl:output omit-xml-declaration="yes"/> <xsl:template match="/"> <books> <xsl:apply-templates select="/bookstore/book/title[../price>30]"> <xsl:sort select="."/> </xsl:apply-templates> </books> </xsl:template> <xsl:template match="node()|@*"> <xsl:copy> <xsl:apply-templates select="node()|@*"/> </xsl:copy> </xsl:template> </xsl:stylesheet> The result will appear as follows: <books> <title lang="en">Apache Camel Developer's Cookbook</title> <title lang="en">Learning XML</title> </books> The Camel XSLT Processor internally runs the message body through a registered Java XML transformer using the XSLT file referenced by the endpoint. This processor uses Camel's Type Converter capabilities to convert the input message body type to one of the supported XML source models in the following order of priority: StAXSource (off by default; this can be enabled by setting allowStAX=true on the endpoint URI) SAXSource StreamSource DOMSource Camel's Type Converter can convert from most input types (String, File, byte[], and so on) to one of the XML source types for most XML content loaded through other Camel endpoints with no extra work on your part. The output data type for the message is, by default, a String, and is configurable using the output parameter on the xslt endpoint URI. There's more... The XSLT Processor passes in headers, properties, and parameters associated with the message. These will show up as XSLT parameters that can be referenced within your XSLT statements. You can pass in the names of the books as parameters to the XSLT template; to do so, modify the previous XLST as follows: <xsl:param name="myParamValue"/> <xsl:template match="/"> <books> <xsl:attribute name="value"> <xsl:value-of select="$myParamValue"/> </xsl:attribute> <xsl:apply-templates select="/bookstore/book/title[../price>$myParamValue]"> <xsl:sort select="."/> </xsl:apply-templates> </books> </xsl:template> The Exchange instance will be associated with a parameter called exchange; the IN message with a parameter called in; and the message headers, properties, and parameters will be associated XSLT parameters with the same name. To use these in your XSLT, you need to explicitly declare a parameter of the same name in your XSLT file. In the previous example, it is possible to use either a message header or exchange property called myParamValue. See also Message Translator: http://camel.apache.org/message-translator.html Camel XSLT Component: http://camel.apache.org/xslt Camel Type Converter: http://camel.apache.org/type-converter.html XSL working group: http://www.w3.org/Style/XSL/ The Transforming inline with XQuery recipe   Transforming from Java to XML with JAXB Camel's JAXB Component is one of a number of components that can be used to convert your XML data back and forth from Java objects. It provides a Camel Data Format that allows you to use JAXB annotated Java classes, and then marshal (Java to XML) or unmarshal (XML to Java) your data. JAXB is a Java standard for translating between XML data and Java that is used by creating annotated Java classes that bind, or map, to your XML data schema. The framework takes care of the rest. This recipe will show you how to use the JAXB Camel Data Format to convert back and forth from Java to XML. Getting ready The Java code for this recipe is located in the org.camelcookbook.transformation.jaxb package. The Spring XML files are located under src/main/resources/META-INF/spring and prefixed with jaxb. To use Camel's JAXB Component, you need to add a dependency element for the camel-jaxb library, which provides the implementation for the JAXB Data Format. Add the following to the dependencies section of your Maven POM: <dependency> lt;groupId>org.apache.camel</groupId> <artifactId>camel-jaxb</artifactId> <version>${camel-version}</version> lt;/dependency>   How to do it... The main steps for converting between Java and XML are as follows: Given a JAXB annotated model, reference that model within a named Camel Data Format. Use that named Data Format within your Camel route using the marshal and unmarshal DSL statements. Create an annotated Java model using standard JAXB annotations. There are a number of external tools that can automate this creation from existing XML or XSD (XML Schema) files: @XmlAccessorType(XmlAccessType.FIELD) @XmlType(name = "", propOrder = { "title", "author", "year", "price" } ) @XmlRootElement(name = "book") public class Book { @XmlElement(required = true) protected Book.Title title; @XmlElement(required = true) protected List<String> author; protected int year; protected double price; // getters and setters } Instantiate a JAXB Data Format within your Camel route that refers to the Java package(s) containing your JAXB annotated classes. In the XML DSL, this is written as: <camelContext > <dataFormats> <jaxb id="myJaxb" contextPath="org.camelcookbook .transformation.myschema"/> </dataFormats> <!-- route definitions here --> </camelContext> In the Java DSL, the Data Format is defined as: public class JaxbRouteBuilder extends RouteBuilder { @Override public void configure() throws Exception { DataFormat myJaxb= new JaxbDataFormat( "org.camelcookbook.transformation.myschema"); // route definitions here } } Reference the Data Format within your route, choosing marshal(Java to XML) or unmarshal(XML to Java) as appropriate. In the XML DSL, this routing logic is written as: <route> <from uri="direct:unmarshal"/> <unmarshal ref="myJaxb"/> </route> In the Java DSL, this is expressed as: from("direct:unmarshal").unmarshal(myJaxb);   How it works... Using Camel JAXB to translate your XML data back and forth to Java makes it much easier for the Java processors defined later on in your route to do custom message processing. This is useful when the built-in XML translators (for example, XSLT or XQuery) are not enough, or you just want to call existing Java code. Camel JAXB eliminates the boilerplate code from your integration flows by providing a wrapper around the standard JAXB mechanisms for instantiating the Java binding for the XML data. There's more... Camel JAXB works just fine with existing JAXB tooling like the maven-jaxb2-plugin plugin, which can automatically create JAXB-annotated Java classes from an XML Schema (XSD). See also Camel JAXB: http://camel.apache.org/jaxb.html Available Data Formats: http://camel.apache.org/data-format.html JAXB Specification: http://jcp.org/en/jsr/detail?id=222   Transforming from Java to JSON Camel's JSON Component is used when you need to convert your JSON data back and forth from Java. It provides a Camel Data Format that, without any requirement for an annotated Java class, allows you to marshal (Java to JSON) or unmarshal (JSON to Java) your data. There is only one step to using Camel JSON to marshal and unmarshal XML data. Within your Camel route, insert the marshal(Java to JSON), or unmarshal(JSON to Java) statement, and configure it to use the JSON Data Format. This recipe will show you how to use the camel-xstream library to convert from Java to JSON, and back. Getting ready The Java code for this recipe is located in the org.camelcookbook.transformation.json package. The Spring XML files are located under src/main/resources/META-INF/spring and prefixed with json. To use Camel's JSON Component, you need to add a dependency element for the camel-xstream library, which provides an implementation for the JSON Data Format using the XStream library. Add the following to the dependencies section of your Maven POM: <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-xstream</artifactId> <version>${camel-version}</version> </dependency>   How to do it... Reference the Data Format within your route, choosing the marshal (Java to JSON), or unmarshal (JSON to Java) statement, as appropriate: In the XML DSL, this is written as follows: <route> <from uri="direct:marshal"/> <marshal> <json/> </marshal> <to uri="mock:marshalResult"/> </route> In the Java DSL, this same route is expressed as: from("direct:marshal") .marshal().json() .to("mock:marshalResult");   How it works... Using Camel JSON simplifies translating your data between JSON and Java. This is convenient when you are dealing with REST endpoints and need Java processors in Camel to do custom message processing later on in the route. Camel JSON provides a wrapper around the JSON libraries for instantiating the Java binding for the JSON data, eliminating more boilerplate code from your integration flows. There's more... Camel JSON works with the XStream library by default, and can be configured to use other JSON libraries, such as Jackson or GSon. These other libraries provide additional features, more customization, and more flexibility that can be leveraged by Camel. To use them, include their respective Camel components, for example, camel-jackson, and specify the library within the json element: <dataFormats> <json id="myJson" library="Jackson"/> </dataFormats>   See also Camel JSON: http://camel.apache.org/json.html Available Data Formats: http://camel.apache.org/data-format.html   Transforming from XML to JSON Camel provides an XML JSON Component that converts your data back and forth between XML and JSON in a single step, without an intermediate Java object representation. It provides a Camel Data Format that allows you to marshal (XML to JSON), or unmarshal (JSON to XML) your data. This recipe will show you how to use the XML JSON Component to convert from XML to JSON, and back. Getting ready Java code for this recipe is located in the org.camelcookbook.transformation.xmljson package. Spring XML files are located under src/main/resources/META-INF/spring and prefixed with xmljson. To use Camel's XML JSON Component, you need to add a dependency element for the camel-xmljson library, which provides an implementation for the XML JSON Data Format. Add the following to the dependencies section of your Maven POM: <dependency> <groupId>org.apache.camel</groupId> <artifactId>camel-xmljson</artifactId> <version>${camel-version}</version> </dependency>   How to do it... Reference the xmljson Data Format within your route, choosing the marshal(XML to JSON), or unmarshal(JSON to XML) statement, as appropriate: In the XML DSL, this is written as follows: <route> <from uri="direct:marshal"/> <marshal> <xmljson/> </marshal> <to uri="mock:marshalResult"/> </route> In the Java DSL, this same route is expressed as: from("direct:marshal") .marshal().xmljson() .to("mock:marshalResult");   How it works... Using the Camel XML JSON Component simplifies translating your data between XML and JSON, making it convenient to use when you are dealing with REST endpoints. The XML JSON Data Format wraps around the Json-lib library, which provides the core translation capabilities, eliminating more boilerplate code from your integration flows. There's more... You may need to configure XML JSON if you want to fine-tune the output of your transformation. For example, consider the following JSON: [{"@category":"PROGRAMMING","title":{"@lang":"en","#text": "Apache Camel Developer's Cookbook"},"author":[ "Scott Cranton","Jakub Korab"],"year":"2013","price":"49.99"}] This will be converted as follows, by default, which may not be exactly what you want (notice the <a> and <e> elements): <?xml version="1.0" encoding="UTF-8"?> <a> <e category="PROGRAMMING"> <author> <e>Scott Cranton</e> <e>Jakub Korab</e> </author> <price>49.99</price> <title lang="en">Apache Camel Developer's Cookbook</title> <year>2013</year> </e> </a> To configure XML JSON to use <bookstore> as the root element instead of <a>, use <book> for the individual elements instead of <e>, and expand the multiple author values to use a sequence of <author> elements, you would need to tune the configuration of the Data Format before referencing it in your route. In the XML DSL, the definition of the Data Format and the route that uses it is written as follows: <dataFormats> <xmljson id="myXmlJson" rootName="bookstore" elementName="book" expandableProperties="author author"/> </dataFormats> <route> <from uri="direct:unmarshalBookstore"/> <unmarshal ref="myXmlJson"/> <to uri="mock:unmarshalResult"/> </route> In the Java DSL, the same thing is expressed as: XmlJsonDataFormat xmlJsonFormat = new XmlJsonDataFormat(); xmlJsonFormat.setRootName("bookstore"); xmlJsonFormat.setElementName("book"); xmlJsonFormat.setExpandableProperties( Arrays.asList("author", "author")); from("direct:unmarshalBookstore") .unmarshal(xmlJsonFormat) .to("mock:unmarshalBookstoreResult"); This will result in the previous JSON being unmarshalled as follows: <?xml version="1.0" encoding="UTF-8"?> <bookstore> <book category="PROGRAMMING"> <author>Scott Cranton</author> <author>Jakub Korab</author><price>49.99</price> <title lang="en">Apache Camel Developer's Cookbook</title> <year>2013</year> </book> </bookstore>   See also Camel XML JSON: http://camel.apache.org/xmljson.html Available Data Formats: http://camel.apache.org/data-format.html Json-lib: http://json-lib.sourceforge.net   Summary We saw how Apache Camel is flexible in allowing us to transform and convert messages in various formats. This makes Apache Camel an ideal choice for integrating different systems together. Resources for Article: Further resources on this subject: Drools Integration Modules: Spring Framework and Apache Camel [Article] Installing Apache Karaf [Article] Working with AMQP [Article]

If science fiction stories are to be believed, the invention of artificial intelligence inevitably leads to apocalyptic wars between machines and their makers. In the early stages, computers are taught to play simple games of tic-tac-toe and chess. Later, machines are given control of traffic lights and communications, followed by military drones and missiles. The machines' evolution takes an ominous turn once the computers become sentient and learn how to teach themselves. Having no more need for human programmers, humankind is then "deleted." Thankfully, at the time of this writing, machines still require user input. Though your impressions of machine learning may be colored by these mass media depictions, today's algorithms are too application-specific to pose any danger of becoming self-aware. The goal of today's machine learning is not to create an artificial brain, but rather to assist us in making sense of the world's massive data stores. Putting popular misconceptions aside, by the end of this article, you will gain a more nuanced understanding of machine learning. You also will be introduced to the fundamental concepts that define and differentiate the most commonly used machine learning approaches. (For more resources related to this topic, see here.) You will learn: The origins and practical applications of machine learning How computers turn data into knowledge and action How to match a machine learning algorithm to your data The field of machine learning provides a set of algorithms that transform data into actionable knowledge. Keep reading to see how easy it is to use R to start applying machine learning to real-world problems. The origins of machine learning Since birth, we are inundated with data. Our body's sensors—the eyes, ears, nose, tongue, and nerves—are continually assailed with raw data that our brain translates into sights, sounds, smells, tastes, and textures. Using language, we are able to share these experiences with others. From the advent of written language, human observations have been recorded. Hunters monitored the movement of animal herds, early astronomers recorded the alignment of planets and stars, and cities recorded tax payments, births, and deaths. Today, such observations, and many more, are increasingly automated and recorded systematically in the ever-growing computerized databases. The invention of electronic sensors has additionally contributed to an explosion in the volume and richness of recorded data. Specialized sensors see, hear, smell, taste, and feel. These sensors process the data far differently than a human being would. Unlike a human's limited and subjective attention, an electronic sensor never takes a break and never lets its judgment skew its perception. Although sensors are not clouded by subjectivity, they do not necessarily report a single, definitive depiction of reality. Some have an inherent measurement error, due to hardware limitations. Others are limited by their scope. A black and white photograph provides a different depiction of its subject than one shot in color. Similarly, a microscope provides a far different depiction of reality than a telescope. Between databases and sensors, many aspects of our lives are recorded. Governments, businesses, and individuals are recording and reporting information, from the monumental to the mundane. Weather sensors record temperature and pressure data, surveillance cameras watch sidewalks and subway tunnels, and all manner of electronic behaviors are monitored: transactions, communications, friendships, and many others. This deluge of data has led some to state that we have entered an era of Big Data, but this may be a bit of a misnomer. Human beings have always been surrounded by large amounts of data. What makes the current era unique is that we have vast amounts of recorded data, much of which can be directly accessed by computers. Larger and more interesting data sets are increasingly accessible at the tips of our fingers, only a web search away. This wealth of information has the potential to inform action, given a systematic way of making sense from it all. The field of study interested in the development of computer algorithms to transform data into intelligent action is known as machine learning. This field originated in an environment where available data, statistical methods, and computing power rapidly and simultaneously evolved. Growth in data necessitated additional computing power, which in turn spurred the development of statistical methods to analyze large datasets. This created a cycle of advancement, allowing even larger and more interesting data to be collected.   A closely related sibling of machine learning, data mining, is concerned with the generation of novel insights from large databases. As the implies, data mining involves a systematic hunt for nuggets of actionable intelligence. Although there is some disagreement over how widely machine learning and data mining overlap, a potential point of distinction is that machine learning focuses on teaching computers how to use data to solve a problem, while data mining focuses on teaching computers to identify patterns that humans then use to solve a problem. Virtually all data mining involves the use of machine learning, but not all machine learning involves data mining. For example, you might apply machine learning to data mine automobile traffic data for patterns related to accident rates; on the other hand, if the computer is learning how to drive the car itself, this is purely machine learning without data mining. The phrase "data mining" is also sometimes used as a pejorative to describe the deceptive practice of cherry-picking data to support a theory. Uses and abuses of machine learning Most people have heard of the chess-playing computer Deep Blue—the first to win a game against a world champion—or Watson, the computer that defeated two human opponents on the television trivia game show Jeopardy. Based on these stunning accomplishments, some have speculated that computer intelligence will replace humans in many information technology occupations, just as machines replaced humans in the fields, and robots replaced humans on the assembly line. The truth is that even as machines reach such impressive milestones, they are still relatively limited in their ability to thoroughly understand a problem. They are pure intellectual horsepower without direction. A computer may be more capable than a human of finding subtle patterns in large databases, but it still needs a human to motivate the analysis and turn the result into meaningful action. Machines are not good at asking questions, or even knowing what questions to ask. They are much better at answering them, provided the question is stated in a way the computer can comprehend. Present-day machine learning algorithms partner with people much like a bloodhound partners with its trainer; the dog's sense of smell may be many times stronger than its master's, but without being carefully directed, the hound may end up chasing its tail.   To better understand the real-world applications of machine learning, we'll now consider some cases where it has been used successfully, some places where it still has room for improvement, and some situations where it may do more harm than good. Machine learning successes Machine learning is most successful when it augments rather than replaces the specialized knowledge of a subject-matter expert. It works with medical doctors at the forefront of the fight to eradicate cancer, assists engineers and programmers with our efforts to create smarter homes and automobiles, and helps social scientists build knowledge of how societies function. Toward these ends, it is employed in countless businesses, scientific laboratories, hospitals, and governmental organizations. Any organization that generates or aggregates data likely employs at least one machine learning algorithm to help make sense of it. Though it is impossible to list every use case of machine learning, a survey of recent success stories includes several prominent applications: Identification of unwanted spam messages in e-mail Segmentation of customer behavior for targeted advertising Forecasts of weather behavior and long-term climate changes Reduction of fraudulent credit card transactions Actuarial estimates of financial damage of storms and natural disasters Prediction of popular election outcomes Development of algorithms for auto-piloting drones and self-driving cars Optimization of energy use in homes and office buildings Projection of areas where criminal activity is most likely Discovery of genetic sequences linked to diseases The limits of machine learning Although machine learning is used widely and has tremendous potential, it is important to understand its limits. Machine learning, at this time, is not in any way a substitute for a human brain. It has very little flexibility to extrapolate outside of the strict parameters it learned and knows no common sense. With this in mind, one should be extremely careful to recognize exactly what the algorithm has learned before setting it loose in the real-world settings. Without a lifetime of past experiences to build upon, computers are also limited in their ability to make simple common sense inferences about logical next steps. Take, for instance, the banner advertisements seen on many web sites. These may be served, based on the patterns learned by data mining the browsing history of millions of users. According to this data, someone who views the websites selling shoes should see advertisements for shoes, and those viewing websites for mattresses should see advertisements for mattresses. The problem is that this becomes a never-ending cycle in which additional shoe or mattress advertisements are served rather than advertisements for shoelaces and shoe polish, or bed sheets and blankets. Many are familiar with the deficiencies of machine learning's ability to understand or translate language or to recognize speech and handwriting. Perhaps the earliest example of this type of failure is in a 1994 episode of the television show, The Simpsons, which showed a parody of the Apple Newton tablet. For its time, the Newton was known for its state-of-the-art handwriting recognition. Unfortunately for Apple, it would occasionally fail to great effect. The television episode illustrated this through a sequence in which a bully's note to Beat up Martin was misinterpreted by the Newton as Eat up Martha, as depicted in the following screenshots:   Screenshots from "Lisa on Ice" The Simpsons, 20th Century Fox (1994) Machines' ability to understand language has improved enough since 1994, such that Google, Apple, and Microsoft are all confident enough to offer virtual concierge services operated via voice recognition. Still, even these services routinely struggle to answer relatively simple questions. Even more, online translation services sometimes misinterpret sentences that a toddler would readily understand. The predictive text feature on many devices has also led to a number of humorous autocorrect fail sites that illustrate the computer's ability to understand basic language but completely misunderstand context. Some of these mistakes are to be expected, for sure. Language is complicated with multiple layers of text and subtext and even human beings, sometimes, understand the context incorrectly. This said, these types of failures in machines illustrate the important fact that machine learning is only as good as the data it learns from. If the context is not directly implicit in the input data, then just like a human, the computer will have to make its best guess. Machine learning ethics At its core, machine learning is simply a tool that assists us in making sense of the world's complex data. Like any tool, it can be used for good or evil. Machine learning may lead to problems when it is applied so broadly or callously that humans are treated as lab rats, automata, or mindless consumers. A process that may seem harmless may lead to unintended consequences when automated by an emotionless computer. For this reason, those using machine learning or data mining would be remiss not to consider the ethical implications of the art. Due to the relative youth of machine learning as a discipline and the speed at which it is progressing, the associated legal issues and social norms are often quite uncertain and constantly in flux. Caution should be exercised while obtaining or analyzing data in order to avoid breaking laws, violating terms of service or data use agreements, and abusing the trust or violating the privacy of customers or the public. The informal corporate motto of Google, an organization that collects perhaps more data on individuals than any other, is "don't be evil." While this seems clear enough, it may not be sufficient. A better approach may be to follow the Hippocratic Oath, a medical principle that states "above all, do no harm." Retailers routinely use machine learning for advertising, targeted promotions, inventory management, or the layout of the items in the store. Many have even equipped checkout lanes with devices that print coupons for promotions based on the customer's buying history. In exchange for a bit of personal data, the customer receives discounts on the specific products he or she wants to buy. At first, this appears relatively harmless. But consider what happens when this practice is taken a little bit further. One possibly apocryphal tale concerns a large retailer in the U.S. that employed machine learning to identify expectant mothers for coupon mailings. The retailer hoped that if these mothers-to-be received substantial discounts, they would become loyal customers, who would later purchase profitable items like diapers, baby formula, and toys. Equipped with machine learning methods, the retailer identified items in the customer purchase history that could be used to predict with a high degree of certainty, not only whether a woman was pregnant, but also the approximate timing for when the baby was due. After the retailer used this data for a promotional mailing, an angry man contacted the chain and demanded to know why his teenage daughter received coupons for maternity items. He was furious that the retailer seemed to be encouraging teenage pregnancy! As the story goes, when the retail chain's manager called to offer an apology, it was the father that ultimately apologized because, after confronting his daughter, he discovered that she was indeed pregnant! Whether completely true or not, the lesson learned from the preceding tale is that common sense should be applied before blindly applying the results of a machine learning analysis. This is particularly true in cases where sensitive information such as health data is concerned. With a bit more care, the retailer could have foreseen this scenario, and used greater discretion while choosing how to reveal the pattern its machine learning analysis had discovered. Certain jurisdictions may prevent you from using racial, ethnic, religious, or other protected class data for business reasons.  Keep in mind that excluding this data from your analysis may not be enough, because machine learning algorithms might inadvertently learn this information independently. For instance, if a certain segment of people generally live in a certain region, buy a certain product, or otherwise behave in a way that uniquely identifies them as a group, some machine learning algorithms can infer the protected information from these other factors. In such cases, you may need to fully "de-identify" these people by excluding any potentially identifying data in addition to the protected information. Apart from the legal consequences, using data inappropriately may hurt the bottom line. Customers may feel uncomfortable or become spooked if the aspects of their lives they consider private are made public. In recent years, several high-profile web applications have experienced a mass exodus of users who felt exploited when the applications' terms of service agreements changed, and their data was used for purposes beyond what the users had originally agreed upon. The fact that privacy expectations differ by context, age cohort, and locale adds complexity in deciding the appropriate use of personal data. It would be wise to consider the cultural implications of your work before you begin your project. The fact that you can use data for a particular end does not always mean that you should. How machines learn A formal definition of machine learning proposed by computer scientist Tom M. Mitchellstates that a machine learns whenever it is able to utilize its an experience such that its performance improves on similar experiences in the future. Although this definition is intuitive, it completely ignores the process of exactly how experience can be translated into future action—and of course learning is always easier said than done! While human brains are naturally capable of learning from birth, the conditions necessary for computers to learn must be made explicit. For this reason, although it is not strictly necessary to understand the theoretical basis of learning, this foundation helps understand, distinguish, and implement machine learning algorithms. As you compare machine learning to human learning, you may discover yourself examining your own mind in a different light. Regardless of whether the learner is a human or machine, the basic learning process is similar. It can be divided into four interrelated components: Data storage utilizes observation, memory, and recall to provide a factual basis for further reasoning. Abstraction involves the translation of stored data into broader representations and concepts. Generalization uses abstracted data to create knowledge and inferences that drive action in new contexts. Evaluation provides a feedback mechanism to measure the utility of learned knowledge and inform potential improvements.  Keep in mind that although the learning process has been conceptualized as four distinct components, they are merely organized this way for illustrative purposes. In reality, the entire learning process is inextricably linked. In human beings, the process occurs subconsciously. We recollect, deduce, induct, and intuit with the confines of our mind's eye, and because this process is hidden, any differences from person to person are attributed to a vague notion of subjectivity. In contrast, with computers these processes are explicit, and because the entire process is transparent, the learned knowledge can be examined, transferred, and utilized for future action. Data storage All learning must begin with data. Humans and computers alike utilize data storage as a foundation for more advanced reasoning. In a human being, this consists of a brain that uses electrochemical signals in a network of biological cells to store and process observations for short- and long-term future recall. Computers have similar capabilities of short- and long-term recall using hard disk drives, flash memory, and random access memory (RAM) in combination with a central processing unit (CPU). It may seem obvious to say so, but the ability to store and retrieve data alone is not sufficient for learning. Without a higher level of understanding, knowledge is limited exclusively to recall, meaning exclusively what is seen before and nothing else. The data is merely ones and zeros on a disk. They are stored memories with no broader meaning. To better understand the nuances of this idea, it may help to think about the last time you studied for a difficult test, perhaps for a university final exam or a career certification. Did you wish for an eidetic (photographic) memory? If so, you may be disappointed to learn that perfect recall is unlikely to be of much assistance. Even if you could memorize material perfectly, your rote learning is of no use, unless you know in advance the exact questions and answers that will appear in the exam. Otherwise, you would be stuck in an attempt to memorize answers to every question that could conceivably be asked. Obviously, this is an unsustainable strategy. Instead, a better approach is to spend time selectively, memorizing a small set of representative ideas while developing strategies on how the ideas relate and how to use the stored information. In this way, large ideas can be understood without needing to memorize them by rote. Abstraction This work of assigning meaning to stored data occurs during the abstraction process, in which raw data comes to have a more abstract meaning. This type of connection, say between an object and its representation, is exemplified by the famous René Magritte painting The Treachery of Images:   Source: http://collections.lacma.org/node/239578 The painting depicts a tobacco pipe with the caption Ceci n'est pas une pipe ("this is not a pipe"). The point Magritte was illustrating is that a representation of a pipe is not truly a pipe. Yet, in spite of the fact that the pipe is not real, anybody viewing the painting easily recognizes it as a pipe. This suggests that the observer's mind is able to connect the picture of a pipe to the idea of a pipe, to a memory of a physical pipe that could be held in the hand. Abstracted connections like these are the basis of knowledge representation, the formation of logical structures that assist in turning raw sensory information into a meaningful insight. During a machine's process of knowledge representation, the computer summarizes stored raw data using a model, an explicit description of the patterns within the data. Just like Magritte's pipe, the model representation takes on a life beyond the raw data. It represents an idea greater than the sum of its parts. There are many different types of models. You may be already familiar with some. Examples include: Mathematical equations Relational diagrams such as trees and graphs Logical if/else rules Groupings of data known as clusters The choice of model is typically not left up to the machine. Instead, the learning task and data on hand inform model selection. Later in this article, we will discuss methods to choose the type of model in more detail. The process of fitting a model to a dataset is known as training. When the model has been trained, the data is transformed into an abstract form that summarizes the original information. You might wonder why this step is called training rather than learning. First, note that the process of learning does not end with data abstraction; the learner must still generalize and evaluate its training. Second, the word training better connotes the fact that the human teacher trains the machine student to understand the data in a specific way. It is important to note that a learned model does not itself provide new data, yet it does result in new knowledge. How can this be? The answer is that imposing an assumed structure on the underlying data gives insight into the unseen by supposing a concept about how data elements are related. Take for instance the discovery of gravity. By fitting equations to observational data, Sir Isaac Newton inferred the concept of gravity. But the force we now know as gravity was always present. It simply wasn't recognized until Newton recognized it as an abstract concept that relates some data to others—specifically, by becoming the g term in a model that explains observations of falling objects.   Most models may not result in the development of theories that shake up scientific thought for centuries. Still, your model might result in the discovery of previously unseen relationships among data. A model trained on genomic data might find several genes that, when combined, are responsible for the onset of diabetes; banks might discover a seemingly innocuous type of transaction that systematically appears prior to fraudulent activity; and psychologists might identify a combination of personality characteristics indicating a new disorder. These underlying patterns were always present, but by simply presenting information in a different format, a new idea is conceptualized. Generalization The learning process is not complete until the learner is able to use its abstracted knowledge for future action. However, among the countless underlying patterns that might be identified during the abstraction process and the myriad ways to model these patterns, some will be more useful than others. Unless the production of abstractions is limited, the learner will be unable to proceed. It would be stuck where it started—with a large pool of information, but no actionable insight. The term generalization describes the process of turning abstracted knowledge into a form that can be utilized for future action, on tasks that are similar, but not identical, to those it has seen before. Generalization is a somewhat vague process that is a bit difficult to describe. Traditionally, it has been imagined as a search through the entire set of models (that is, theories or inferences) that could be abstracted during training. In other words, if you can imagine a hypothetical set containing every possible theory that could be established from the data, generalization involves the reduction of this set into a manageable number of important findings. In generalization, the learner is tasked with limiting the patterns it discovers to only those that will be most relevant to its future tasks. Generally, it is not feasible to reduce the number of patterns by examining them one-by-one and ranking them by future utility. Instead, machine learning algorithms generally employ shortcuts that reduce the search space more quickly. Toward this end, the algorithm will employ heuristics, which are educated guesses about where to find the most useful inferences. Because heuristics utilize approximations and other rules of thumb, they do not guarantee to find the single best model. However, without taking these shortcuts, finding useful information in a large dataset would be infeasible. Heuristics are routinely used by human beings to quickly generalize experience to new scenarios. If you have ever utilized your gut instinct to make a snap decision prior to fully evaluating your circumstances, you were intuitively using mental heuristics. The incredible human ability to make quick decisions often relies not on computer-like logic, but rather on heuristics guided by emotions. Sometimes, this can result in illogical conclusions. For example, more people express fear of airline travel versus automobile travel, despite automobiles being statistically more dangerous. This can be explained by the availability heuristic, which is the tendency of people to estimate the likelihood of an event by how easily its examples can be recalled. Accidents involving air travel are highly publicized. Being traumatic events, they are likely to be recalled very easily, whereas car accidents barely warrant a mention in the newspaper. The folly of misapplied heuristics is not limited to human beings. The heuristics employed by machine learning algorithms also sometimes result in erroneous conclusions. The algorithm is said to have a bias if the conclusions are systematically erroneous, or wrong in a predictable manner. For example, suppose that a machine learning algorithm learned to identify faces by finding two dark circles representing eyes, positioned above a straight line indicating a mouth. The algorithm might then have trouble with, or be biased against, faces that do not conform to its model. Faces with glasses, turned at an angle, looking sideways, or with various skin tones might not be detected by the algorithm. Similarly, it could be biased toward faces with certain skin tones, face shapes, or other characteristics that do not conform to its understanding of the world.   In modern usage, the word bias has come to carry quite negative connotations. Various forms of media frequently claim to be free from bias, and claim to report the facts objectively, untainted by emotion. Still, consider for a moment the possibility that a little bias might be useful. Without a bit of arbitrariness, might it be a bit difficult to decide among several competing choices, each with distinct strengths and weaknesses? Indeed, some recent studies in the field of psychology have suggested that individuals born with damage to portions of the brain responsible for emotion are ineffectual in decision making, and might spend hours debating simple decisions such as what color shirt to wear or where to eat lunch. Paradoxically, bias is what blinds us from some information while also allowing us to utilize other information for action. It is how machine learning algorithms choose among the countless ways to understand a set of data. Evaluation Bias is a necessary evil associated with the abstraction and generalization processes inherent in any learning task. In order to drive action in the face of limitless possibility, each learner must be biased in a particular way. Consequently, each learner has its weaknesses and there is no single learning algorithm to rule them all. Therefore, the final step in the generalization process is to evaluate or measure the learner's success in spite of its biases and use this information to inform additional training if needed. Once you've had success with one machine learning technique, you might be tempted to apply it to everything. It is important to resist this temptation because no machine learning approach is the best for every circumstance. This fact is described by the No Free Lunch theorem, introduced by David Wolpert in 1996. For more information, visit: http://www.no-free-lunch.org. Generally, evaluation occurs after a model has been trained on an initial training dataset. Then, the model is evaluated on a new test dataset in order to judge how well its characterization of the training data generalizes to new, unseen data. It's worth noting that it is exceedingly rare for a model to perfectly generalize to every unforeseen case. In parts, models fail to perfectly generalize due to the problem of noise, a term that describes unexplained or unexplainable variations in data. Noisy data is caused by seemingly random events, such as: Measurement error due to imprecise sensors that sometimes add or subtract a bit from the readings Issues with human subjects, such as survey respondents reporting random answers to survey questions, in order to finish more quickly Data quality problems, including missing, null, truncated, incorrectly coded, or corrupted values Phenomena that are so complex or so little understood that they impact the data in ways that appear to be unsystematic Trying to model noise is the basis of a problem called overfitting. Because most noisy data is unexplainable by definition, attempting to explain the noise will result in erroneous conclusions that do not generalize well to new cases. Efforts to explain the noise will also typically result in more complex models that will miss the true pattern that the learner tries to identify. A model that seems to perform well during training, but does poorly during evaluation, is said to be overfitted to the training dataset, as it does not generalize well to the test dataset.   Solutions to the problem of overfitting are specific to particular machine learning approaches. For now, the important point is to be aware of the issue. How well the models are able to handle noisy data is an important source of distinction among them. Machine learning in practice So far, we've focused on how machine learning works in theory. To apply the learning process to real-world tasks, we'll use a five-step process. Regardless of the task at hand, any machine learning algorithm can be deployed by following these steps: Data collection: The data collection step involves gathering the learning material an algorithm will use to generate actionable knowledge. In most cases, the data will need to be combined into a single source like a text file, spreadsheet, or database. Data exploration and preparation: The quality of any machine learning project is based largely on the quality of its input data. Thus, it is important to learn more about the data and its nuances during a practice called data exploration. Additional work is required to prepare the data for the learning process. This involves fixing or cleaning so-called "messy" data, eliminating unnecessary data, and recoding the data to conform to the learner's expected inputs. Model training: By the time the data has been prepared for analysis, you are likely to have a sense of what you are capable of learning from the data. The specific machine learning task chosen will inform the selection of an appropriate algorithm, and the algorithm will represent the data in the form of a model. Model evaluation: Because each machine learning model results in a biased solution to the learning problem, it is important to evaluate how well the algorithm learns from its experience. Depending on the type of model used, you might be able to evaluate the accuracy of the model using a test dataset or you may need to develop measures of performance specific to the intended application. Model improvement: If better performance is needed, it becomes necessary to utilize more advanced strategies to augment the performance of the model. Sometimes, it may be necessary to switch to a different type of model altogether. You may need to supplement your data with additional data or perform additional preparatory work as in step two of this process. After these steps are completed, if the model appears to be performing well, it can be deployed for its intended task. As the case may be, you might utilize your model to provide score data for predictions (possibly in real time), for projections of financial data, to generate useful insight for marketing or research, or to automate tasks such as mail delivery or flying aircraft. The successes and failures of the deployed model might even provide additional data to train your next generation learner. Types of input data The practice of machine learning involves matching the characteristics of input data to the biases of the available approaches. Thus, before applying machine learning to real-world problems, it is important to understand the terminology that distinguishes among input datasets. The phrase unit of observation is used to describe the smallest entity with measured properties of interest for a study. Commonly, the unit of observation is in the form of persons, objects or things, transactions, time points, geographic regions, or measurements. Sometimes, units of observation are combined to form units such as person-years, which denote cases where the same person is tracked over multiple years; each person-year comprises of a person's data for one year. The unit of observation is related, but not identical, to the unit of analysis, which is the smallest unit from which the inference is made. Although it is often the case, the observed and analyzed units are not always the same. For example, data observed from people might be used to analyze trends across countries. Datasets that store the units of observation and their properties can be imagined as collections of data consisting of: Examples: Instances of the unit of observation for which properties have been recorded Features: Recorded properties or attributes of examples that may be useful for learning It is easiest to understand features and examples through real-world cases. To build a learning algorithm to identify spam e-mail, the unit of observation could be e-mail messages, the examples would be specific messages, and the features might consist of the words used in the messages. For a cancer detection algorithm, the unit of observation could be patients, the examples might include a random sample of cancer patients, and the features may be the genomic markers from biopsied cells as well as the characteristics of patient such as weight, height, or blood pressure. While examples and features do not have to be collected in any specific form, they are commonly gathered in matrix format, which means that each example has exactly the same features. The following spreadsheet shows a dataset in matrix format. In matrix data, each row in the spreadsheet is an example and each column is a feature. Here, the rows indicate examples of automobiles, while the columns record various each automobile's features, such as price, mileage, color, and transmission type. Matrix format data is by far the most common form used in machine learning: Features also come in various forms. If a feature represents a characteristic measured in numbers, it is unsurprisingly called numeric. Alternatively, if a feature is an attribute that consists of a set of categories, the feature is called categorical or nominal. A special case of categorical variables is called ordinal, which designates a nominal variable with categories falling in an ordered list. Some examples of ordinal variables include clothing sizes such as small, medium, and large; or a measurement of customer satisfaction on a scale from "not at all happy" to "very happy." It is important to consider what the features represent, as the type and number of features in your dataset will assist in determining an appropriate machine learning algorithm for your task. Types of machine learning algorithms Machine learning algorithms are divided into categories according to their purpose. Understanding the categories of learning algorithms is an essential first step towards using data to drive the desired action. A predictive model is used for tasks that involve, as the name implies, the prediction of one value using other values in the dataset. The learning algorithm attempts to discover and model the relationship between the target feature (the feature being predicted) and the other features. Despite the common use of the word "prediction" to imply forecasting, predictive models need not necessarily foresee events in the future. For instance, a predictive model could be used to predict past events, such as the date of a baby's conception using the mother's present-day hormone levels. Predictive models can also be used in real time to control traffic lights during rush hours. Because predictive models are given clear instruction on what they need to learn and how they are intended to learn it, the process of training a predictive model is known as supervised learning. The supervision does not refer to human involvement, but rather to the fact that the target values provide a way for the learner to know how well it has learned the desired task. Stated more formally, given a set of data, a supervised learning algorithm attempts to optimize a function (the model) to find the combination of feature values that result in the target output. The often used supervised machine learning task of predicting which category an example belongs to is known as classification. It is easy to think of potential uses for a classifier. For instance, you could predict whether: An e-mail message is spam A person has cancer A football team will win or lose An applicant will default on a loan In classification, the target feature to be predicted is a categorical feature known as the class, and is divided into categories called levels. A class can have two or more levels, and the levels may or may not be ordinal. Because classification is so widely used in machine learning, there are many types of classification algorithms, with strengths and weaknesses suited for different types of input data. Supervised learners can also be used to predict numeric data such as income, laboratory values, test scores, or counts of items. To predict such numeric values, a common form of numeric prediction fits linear regression models to the input data. Although regression models are not the only type of numeric models, they are, by far, the most widely used. Regression methods are widely used for forecasting, as they quantify in exact terms the association between inputs and the target, including both, the magnitude and uncertainty of the relationship. Since it is easy to convert numbers into categories (for example, ages 13 to 19 are teenagers) and categories into numbers (for example, assign 1 to all males, 0 to all females), the boundary between classification models and numeric prediction models is not necessarily firm. A descriptive model is used for tasks that would benefit from the insight gained from summarizing data in new and interesting ways. As opposed to predictive models that predict a target of interest, in a descriptive model, no single feature is more important than any other. In fact, because there is no target to learn, the process of training a descriptive model is called unsupervised learning. Although it can be more difficult to think of applications for descriptive models—after all, what good is a learner that isn't learning anything in particular—they are used quite regularly for data mining. For example, the descriptive modeling task called pattern discovery is used to identify useful associations within data. Pattern discovery is often used for market basket analysis on retailers' transactional purchase data. Here, the goal is to identify items that are frequently purchased together, such that the learned information can be used to refine marketing tactics. For instance, if a retailer learns that swimming trunks are commonly purchased at the same time as sunglasses, the retailer might reposition the items more closely in the store or run a promotion to "up-sell" customers on associated items. Originally used only in retail contexts, pattern discovery is now starting to be used in quite innovative ways. For instance, it can be used to detect patterns of fraudulent behavior, screen for genetic defects, or identify hot spots for criminal activity. The descriptive modeling task of dividing a dataset into homogeneous groups is called clustering. This is sometimes used for segmentation analysis that identifies groups of individuals with similar behavior or demographic information, so that advertising campaigns could be tailored for particular audiences. Although the machine is capable of identifying the clusters, human intervention is required to interpret them. For example, given five different clusters of shoppers at a grocery store, the marketing team will need to understand the differences among the groups in order to create a promotion that best suits each group. Lastly, a class of machine learning algorithms known as meta-learners is not tied to a specific learning task, but is rather focused on learning how to learn more effectively. A meta-learning algorithm uses the result of some learnings to inform additional learning. This can be beneficial for very challenging problems or when a predictive algorithm's performance needs to be as accurate as possible. Machine learning with R Many of the algorithms needed for machine learning with R are not included as part of the base installation. Instead, the algorithms needed for machine learning are available via a large community of experts who have shared their work freely. These must be installed on top of base R manually. Thanks to R's status as free open source software, there is no additional charge for this functionality. A collection of R functions that can be shared among users is called a package. Free packages exist for each of the machine learning algorithms covered in this book. In fact, this book only covers a small portion of all of R's machine learning packages. If you are interested in the breadth of R packages, you can view a list at Comprehensive R Archive Network (CRAN), a collection of web and FTP sites located around the world to provide the most up-to-date versions of R software and packages. If you obtained the R software via download, it was most likely from CRAN at http://cran.r-project.org/index.html. If you do not already have R, the CRAN website also provides installation instructions and information on where to find help if you have trouble. The Packages link on the left side of the page will take you to a page where you can browse packages in an alphabetical order or sorted by the publication date. At the time of writing this, a total 6,779 packages were available—a jump of over 60 percent in the time since the first edition was written, and this trend shows no sign of slowing! The Task Views link on the left side of the CRAN page provides a curated list of packages as per the subject area. The task view for machine learning, which lists the packages covered in this book (and many more), is available at http://cran.r-project.org/web/views/MachineLearning.html. Installing R packages Despite the vast set of available R add-ons, the package format makes installation and use a virtually effortless process. To demonstrate the use of packages, we will install and load the RWeka package, which was developed by Kurt Hornik, Christian Buchta, and Achim Zeileis (see Open-Source Machine Learning: R Meets Weka in Computational Statistics 24: 225-232 for more information). The RWeka package provides a collection of functions that give R access to the machine learning algorithms in the Java-based Weka software package by Ian H. Witten and Eibe Frank. More information on Weka is available at http://www.cs.waikato.ac.nz/~ml/weka/ To use the RWeka package, you will need to have Java installed (many computers come with Java preinstalled). Java is a set of programming tools available for free, which allow for the use of cross-platform applications such as Weka. For more information, and to download Java on your system, you can visit http://java.com. The most direct way to install a package is via the install.packages() function. To install the RWeka package, at the R command prompt, simply type: > install.packages("RWeka") R will then connect to CRAN and download the package in the correct format for your OS. Some packages such as RWeka require additional packages to be installed before they can be used (these are called dependencies). By default, the installer will automatically download and install any dependencies. The first time you install a package, R may ask you to choose a CRAN mirror. If this happens, choose the mirror residing at a location close to you. This will generally provide the fastest download speed. The default installation options are appropriate for most systems. However, in some cases, you may want to install a package to another location. For example, if you do not have root or administrator privileges on your system, you may need to specify an alternative installation path. This can be accomplished using the lib option, as follows: > install.packages("RWeka", lib="/path/to/library") The installation function also provides additional options for installation from a local file, installation from source, or using experimental versions. You can read about these options in the help file, by using the following command: > ?install.packages More generally, the question mark operator can be used to obtain help on any R function. Simply type ? before the name of the function. Loading and unloading R packages In order to conserve memory, R does not load every installed package by default. Instead, packages are loaded by users as they are needed, using the library() function. The name of this function leads some people to incorrectly use the terms library and package interchangeably. However, to be precise, a library refers to the location where packages are installed and never to a package itself. To load the RWeka package we installed previously, you can type the following: > library(RWeka) To unload an R package, use the detach() function. For example, to unload the RWeka package shown previously use the following command: > detach("package:RWeka", unload = TRUE) This will free up any resources used by the package. Summary To learn more about Machine Learning, the following books published by Packt Publishing (https://www.packtpub.com/) are recommended: Machine Learning with R - Second Edition Machine Learning with R Cookbook  Mastering Machine Learning with R  Learning Data Mining with R Resources for Article: Further resources on this subject: Getting Started with RStudio [article] Machine learning and Python – the Dream Team [article] Deep learning in R [article]

A robust penetration testing methodology needs a roadmap. This will provide practical ideas and proven practices which should be handled with great care in order to assess the system security correctly. Let's take a look at what this methodology looks like. It will help ensure you're using BackTrack effectively and that you're tests are thorough and reliable. Penetration testing can be carried out independently or as a part of an IT security risk management process that may be incorporated into a regular development lifecycle (for example, Microsoft SDLC). It is vital to notice that the security of a product not only depends on the factors relating to the IT environment, but also relies on product specific security's best practices. This involves implementation of appropriate security requirements, performing risk analysis, threat modeling, code reviews, and operational security measurement. PenTesting is considered to be the last and most aggressive form of security assessment handled by qualified professionals with or without prior knowledge of a system under examination. It can be used to assess all the IT infrastructure components including applications, network devices, operating systems, communication medium, physical security, and human psychology. The output of penetration testing usually contains a report which is divided into several sections addressing the weaknesses found in the current state of a system following their countermeasures and recommendations. Thus, the use of a methodological process provides extensive benefits to the pentester to understand and critically analyze the integrity of current defenses during each stage of the testing process. Different types of penetration testing Although there are different types of penetration testing, the two most general approaches that are widely accepted by the industry are Black-Box and White-Box. These approaches will be discussed in the following sections. Black-box testing The black-box approach is also known as external testing. While applying this approach, the security auditor will be assessing the network infrastructure from a remote location and will not be aware of any internal technologies deployed by the concerning organization. By employing the number of real world hacker techniques and following through organized test phases, it may reveal some known and unknown set of vulnerabilities which may otherwise exist on the network. An auditor dealing with black-box testing is also known as black-hat. It is important for an auditor to understand and classify these vulnerabilities according to their level of risk (low, medium, or high). The risk in general can be measured according to the threat imposed by the vulnerability and the financial loss that would have occurred following a successful penetration. An ideal penetration tester would undermine any possible information that could lead him to compromise his target. Once the test process is completed, a report is generated with all the necessary information regarding the target security assessment, categorizing and translating the identified risks into business context. White-box testing The white-box approach is also referred to as internal testing. An auditor involved in this kind of penetration testing process should be aware of all the internal and underlying technologies used by the target environment. Hence, it opens a wide gate for an auditor to view and critically evaluate the security vulnerabilities with minimum possible efforts. An auditor engaged with white-box testing is also known as white-hat. It does bring more value to the organization as compared to the blackbox approach in the sense that it will eliminate any internal security issues lying at the target infrastructure environment, thus, making it more tightened for malicious adversary to infiltrate from the outside. The number of steps involved in white-box testing is a bit more similar to that of black-box, except the use of the target scoping, information gathering, and identification phases can be excluded. Moreover, the white-box approach can easily be integrated into a regular development lifecycle to eradicate any possible security issues at its early stage before they get disclosed and exploited by intruders. The time and cost required to find and resolve the security vulnerabilities is comparably less than the black-box approach. The combination of both types of penetration testing provides a powerful insight for internal and external security viewpoints. This combination is known as Grey-Box testing, and the auditor engaged with gray-box testing is also known as grey-hat. The key benefit in devising and practicing a gray-box approach is a set of advantages posed by both approaches mentioned earlier. However, it does require an auditor with limited knowledge of an internal system to choose the best way to assess its overall security. On the other side, the external testing scenarios geared by the graybox approach are similar to that of the black-box approach itself, but can help in making better decisions and test choices because the auditor is informed and aware of the underlying technology. Vulnerability assessment versus penetration testing Since the exponential growth of an IT security industry, there are always an intensive number of diversities found in understanding and practicing the correct terminology for security assessment. This involves commercial grade companies and non-commercial organizations who always misinterpret the term while contracting for the specific type of security assessment. For this obvious reason, we decided to include a brief description on vulnerability assessment and differentiate its core features with penetration testing. Vulnerability assessment is a process for assessing the internal and external security controls by identifying the threats that pose serious exposure to the organizations assets. This technical infrastructure evaluation not only points the risks in the existing defenses but also recommends and prioritizes the remediation strategies. The internal vulnerability assessment provides an assurance for securing the internal systems, while the external vulnerability assessment demonstrates the security of the perimeter defenses. In both testing criteria, each asset on the network is rigorously tested against multiple attack vectors to identify unattended threats and quantify the reactive measures. Depending on the type of assessment being carried out, a unique set of testing process, tools, and techniques are followed to detect and identify vulnerabilities in the information assets in an automated fashion. This can be achieved by using an integrated vulnerability management platform that manages an up-to-date vulnerabilities database and is capable of testing different types of network devices while maintaining the integrity of configuration and change management. A key difference between vulnerability assessment and penetration testing is that penetration testing goes beyond the level of identifying vulnerabilities and hooks into the process of exploitation, privilege escalation, and maintaining access to the target system. On the other hand, vulnerability assessment provides a broad view of any existing flaws in the system without measuring the impact of these flaws to the system under consideration. Another major difference between both of these terms is that the penetration testing is considerably more intrusive than vulnerability assessment and aggressively applies all the technical methods to exploit the live production environment. However, the vulnerability assessment process carefully identifies and quantifies all the vulnerabilities in a non-invasive manner. This perception of an industry, while dealing with both of these assessment types, may confuse and overlap the terms interchangeably, which is absolutely wrong. A qualified consultant always makes an exception to workout the best type of assessment based on the client's business requirement rather than misleading them from one over the other. It is also a duty of the contracting party to look into the core details of the selected security assessment program before taking any final decision. Penetration testing is an expensive service when compared to vulnerability assessment. Security testing methodologies There have been various open source methodologies introduced to address security assessment needs. Using these assessment methodologies, one can easily pass the time-critical and challenging task of assessing the system security depending on its size and complexity. Some of these methodologies focus on the technical aspect of security testing, while others focus on managerial criteria, and very few address both sides. The basic idea behind formalizing these methodologies with your assessment is to execute different types of tests step-by-step in order to judge the security of a system accurately. Therefore, we have introduced four such well-known security assessment methodologies to provide an extended view of assessing the network and application security by highlighting their key features and benefits. These include: Open Source Security Testing Methodology Manual (OSSTMM) Information Systems Security Assessment Framework (ISSAF) Open Web Application Security Project (OWASP) Top Ten Web Application Security Consortium Threat Classification (WASC-TC) All of these testing frameworks and methodologies will assist the security professionals to choose the best strategy that could fit into their client's requirements and qualify the suitable testing prototype. The first two provide general guidelines and methods adhering security testing for almost any information assets. The last two mainly deal with the assessment of an application security domain. It is, however, important to note that the security in itself is an on-going process. Any minor change in the target environment can affect the whole process of security testing and may introduce errors in the final results. Thus, before complementing any of the above testing methods, the integrity of the target environment should be assured. Additionally, adapting any single methodology does not necessarily provide a complete picture of the risk assessment process. Hence, it is left up to the security auditor to select the best strategy that can address the target testing criteria and remains consistent with its network or application environment. There are many security testing methodologies which claim to be perfect in finding all security issues, but choosing the best one still requires a careful selection process under which one can determine the accountability, cost, and effectiveness of the assessment at optimum level. Thus, determining the right assessment strategy depends on several factors, including the technical details provided about the target environment, resource availability, PenTester's knowledge, business objectives, and regulatory concerns. From a business standpoint, investing blind capital and serving unwanted resources to a security testing process can put the whole business economy in danger. Open Source Security Testing Methodology Manual (OSSTMM) The OSSTMM is a recognized international standard for security testing and analysis and is being used by many organizations in their day-to-day assessment cycle. It is purely based on scientific method which assists in quantifying the operational security and its cost requirements in concern with the business objectives. From a technical perspective, its methodology is divided into four key groups, that is, Scope, Channel, Index, and Vector. The scope defines a process of collecting information on all assets operating in the target environment. A channel determines the type of communication and interaction with these assets, which can be physical, spectrum, and communication. All of these channels depict a unique set of security components that has to be tested and verified during the assessment period. These components comprise of physical security, human psychology, data networks, wireless communication medium, and telecommunication. The index is a method which is considerably useful while classifying these target assets corresponding to their particular identifications, such as, MAC Address, and IP Address. At the end, a vector concludes the direction by which an auditor can assess and analyze each functional asset. This whole process initiates a technical roadmap towards evaluating the target environment thoroughly and is known as Audit Scope. There are different forms of security testing which have been classified under OSSTMM methodology and their organization is presented within six standard security test types: Blind: The blind testing does not require any prior knowledge about the target system. But the target is informed before the execution of an audit scope. Ethical hacking and war gaming are examples of blind type testing. This kind of testing is also widely accepted because of its ethical vision of informing a target in advance. Double blind: In double blind testing, an auditor does not require any knowledge about the target system nor is the target informed before the test execution. Black-box auditing and penetration testing are examples of double blind testing. Most of the security assessments today are carried out using this strategy, thus, putting a real challenge for auditors to select the best of breed tools and techniques in order to achieve their required goal. Gray box: In gray box testing, an auditor holds limited knowledge about the target system and the target is also informed before the test is executed. Vulnerability assessment is one of the basic examples of gray box testing. Double gray box: The double gray box testing works in a similar way to gray box testing, except the time frame for an audit is defined and there are no channels and vectors being tested. White-box audit is an example of double gray box testing. Tandem: In tandem testing, the auditor holds minimum knowledge to assess the target system and the target is also notified in advance before the test is executed. It is fairly noted that the tandem testing is conducted thoroughly. Crystal box and in-house audit are examples of tandem testing. Reversal: In reversal testing, an auditor holds full knowledge about the target system and the target will never be informed of how and when the test will be conducted. Red-teaming is an example of reversal type testing. Which OSSTMM test type follows the rules of Penetration Testing? Double blind testing The technical assessment framework provided by OSSTMM is flexible and capable of deriving certain test cases which are logically divided into five security components of three consecutive channels, as mentioned previously. These test cases generally examine the target by assessing its access control security, process security, data controls, physical location, perimeter protection, security awareness level, trust level, fraud control protection, and many other procedures. The overall testing procedures focus on what has to be tested, how it should be tested, what tactics should be applied before, during and after the test, and how to interpret and correlate the final results. Capturing the current state of protection of a target system by using security metrics is considerably useful and invaluable. Thus, the OSSTMM methodology has introduced this terminology in the form of RAV (Risk Assessment Values). The basic function of RAV is to analyze the test results and compute the actual security value based on three factors, which are operational security, loss controls, and limitations. This final security value is known as RAV Score. By using RAV score an auditor can easily extract and define the milestones based on the current security posture to accomplish better protection. From a business perspective, RAV can optimize the amount of investment required on security and may help in the justification of better available solutions. Key features and benefits Practicing the OSSTMM methodology substantially reduces the occurrence of false negatives and false positives and provides accurate measurement for the security. Its framework is adaptable to many types of security tests, such as penetration testing, white-box audit, vulnerability assessment, and so forth. It ensures the assessment should be carried out thoroughly and that of the results can be aggregated into consistent, quantifiable, and reliable manner. The methodology itself follows a process of four individually connected phases, namely definition phase, information phase, regulatory phase, and controls test phase. Each of which obtain, assess, and verify the information regarding the target environment. Evaluating security metrics can be achieved using the RAV method. The RAV calculates the actual security value based on operational security, loss controls, and limitations. The given output known as the RAV score represents the current state of target security. Formalizing the assessment report using the Security Test Audit Report (STAR) template can be advantageous to management, as well as the technical team to review the testing objectives, risk assessment values, and the output from each test phase. The methodology is regularly updated with new trends of security testing, regulations, and ethical concerns. The OSSTMM process can easily be coordinated with industry regulations, business policy, and government legislations. Additionally, a certified audit can also be eligible for accreditation from ISECOM (Institute for Security and Open Methodologies) directly.

In this tutorial, we will be covering JavaScript from a browser perspective. We will create a single page web app that will show the weather forecast for seven days from the current date. The user will provide a ZIP code as input for which the weather will be displayed. We will display all the basic information about the weather on a given day. We believe in learning by doing practicals. Let's see the power of Kotlin from a browser perspective. [box type="shadow" align="" class="" width=""]This article is an excerpt from the book,  Kotlin Blueprints, written by Ashish Belagali, Hardik Trivedi, and Akshay Chordiya. This book is a practical guide to building industry-grade web, mobile, and desktop applications in Kotlin using frameworks such as Spring Boot and Node.js[/box] Conceptually, we will cover the following points while making a web app: Setting up a project to use Kotlin along with JavaScript Showing simple text using Kotlin code Interacting with Document Object Model (DOM) using Kotlin DSL and usage of kotlinx.html Creating your first Kotlin and JavaScript project Tighten your shoelaces! As a first step, we will do the setup and create a simple app that prints on a console and changes the background color of a page. Choosing an IDE From Microsoft Visual Studio, NetBeans to Eclipse and Code::Blocks, we have a series of great and powerful IDEs. Each of them has their own pros and cons. JetBrains is one of the giants that is famous for its cutting-edge software and IntelliJ IDEA Ultimate is considered among one of the most intelligent IDEs for Java. It supports Kotlin and JavaScript by default. There is no other hassle in setting up the environments. Just install it from https://www.jetbrains.com/idea and you are all set to create your first JavaScript project using Kotlin. Creating a project If you are all done with setting up an IDE, launch IntelliJ IDEA and select Create New Project. You will then have the following screen opened. Select Kotlin | Kotlin (JavaScript) options as shown in the following screenshot: Make sure you select Kotlin (JavaScript) as highlighted in the preceding screenshot. The next step is to provide your Project name and choose a destination for your project directory: Creating an HTML page No browser project is complete without an HTML page. Create an index.html page in the root directory of your project. And write the following lines in a <body> tag: <body> <script type="text/javascript" src="out/production/KotlinWeb/lib/kotlin.js"></script> <script type="text/javascript" src="out/production/KotlinWeb/KotlinWeb.js"></script> </body> Creating a Main.kt file After creating our index.html page. Let's create our first Kotlin file. Name it as Main.kt or provide any desired name. Create a file in the src folder and write the following function inside: fun main(args: Array<String>) { document.bgColor="FF0000" val message = "Kotlin Blueprints" println("Your first JS code using Kotlin") } Build the project, by selecting the Build | Build Project menu option. On expanding the project explorer on the left of your workspace you will have the following type of directory structure: Make sure you double-check that the <script> tags are added in the <body>. They should match the name with the files created inside out/production/KotlinBluePrintsJSDemo/. Running the project If you have followed all the steps simply execute your index.html file in any browser and you should see the following output on your console and a red colored page rendered on your DOM: Congratulations! You have executed your first Kotlin code on the browser. Since we have code written in Kotlin, source code needs to recompile every time we update the code. Simply reloading an HTML page will not work. So build your project from the Build | Build Project menu option. Developing a weather forecast web app It was fun writing Kotlin code for a browser and seeing it working, wasn't it? Now we should target bigger goals. Let's develop another app step by step. We will build a weather forecast app, where the user will enter a ZIP code and can see the weather details (seven-day forecast) for the provided region. We will use the OpenWeatherMap API to get the weather details. Please find more details at https://openweathermap.org/api. Before we move to the next step we should create a new project named KotlinBluePrintsJSDemo. Some quick steps to follow: Create a Kotlin+JavaScript project named KotlinBluePrintsJSDemo. Create an index.html page under the root directory. Create a Main.kt file inside the src directory. Add script tags to add two JavaScript files, kotlin.js and KotlinBluePrintsJSDemo.js. Build a project. We want to create an app that will look like this at the end. Entirely in Kotlin: Creating a UI with dummy data The very first thing we do is to create a dummy view and get a clear idea of how our HTML page will look. We will also use a bit of CSS to give basic styles to our <div> tags. Simple HTML approach Now we shall look at the index.html file that we created by writing the following code. It's boring plain HTML tags: <!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8"> <title>Kotlin BluePrints JS Demo</title> </head> <body> <link rel="stylesheet" type="text/css" href="css/main.css"> <div id="container"> <label>Enter zip code : <input id="zipCode" type="number"> </label> <button id="submitZipCode" type="button">Get Weather</button> <div class="weatherContainer"> <div class="weatherBlock"> <div>13 Oct, 2017</div> <img src="images/weather_img.png" height="40px" width="40px"> <div> <span>35</span> <span>20</span> </div> </div> <div class="weatherBlock"> <div>13 Oct, 2017</div> <img src="images/weather_img.png" height="40px" width="40px"> <div> <span>35</span> <span>20</span> </div> </div> <!-- Similarly you can have remaining divs here --> </div> </div> <script src="out/production/KotlinBluePrintsJSDemo/lib/kotlin.js"> </script> <script src="out/production/KotlinBluePrintsJSDemo /KotlinBluePrintsJSDemo.js"></script> </body> </html> Observe two tags, <script> and <link>. We haven't added CSS yet. Let's create a CSS folder under the root directory and create a main.css file inside. The main.css will contain the following code for now: .weatherContainer { width: 90%; background: #EEEEEE; margin: 10px auto; position: relative; text-align:center; } .weatherBlock { background: #FFFFFF; height: 100px; width: 100px; display:inline-block; margin: 10px; } In a source code, we have also created an images directory and put some weather images in it to make the UI more beautiful. Creating UI using Kotlin The index.html page contains all the HTML code. We need to now move that HTML code to Kotlin. Kotlin has the capability to manipulate the DOM element and it can also deal with the tree elements and their hierarchy. Simply put two <script> tags and a parent <div> tag in an HTML page and everything will go to a Kotlin page: <div id="container"> </div> Now, in Main.kt we will write the HTML code that we previously wrote inside index.html. Main.kt and it will look as follows: fun main(args: Array<String>) { createUserInput() } fun createUserInput() { val root = document.getElementById("container") root?.innerHTML = "<label>Enter zip code : <input id="zipCode" type="number"></label>" + "<button id="submitZipCode" type="button">Get Weather</button>" + "<div class="weatherContainer">" + "<div class="weatherBlock">" + "<div>13 Oct, 2017</div>" + "<img src="images/weather_img.png" height="40px" width="40px">"+ "<div>" + "<span>35</span>" + "<span>20</span>" + "</div>" + "</div>" + "<div class="weatherBlock">" + "<div>13 Oct, 2017</div>" + "<img src="images/weather_img.png" height="40px" width="40px">"+ "<div>" + "<span>35</span>" + "<span>20</span>" + "</div>" + "</div>" // Similarly add remaining divs } Take a note of the document object and its function getElementById. This is coming from the kotlin.browser.document package. Also org.w3c.dom as companion classes for all HTML elements. With object root, we get access to an innerHTML property and we can assign any valid HTML strings to it and it will get rendered. It is noteworthy that the nullability of root objects is handled with Null Safety operator ? of Kotlin. What is DSL? Now, the previous approach doesn't create much difference. Kotlin would want to do better! Let us introduce you to the beautiful concept of DSL. DSL stands for Domain Specific Language. As the name indicates, it gives you the feeling that you are writing code in a language, using terminology particular to a given domain, without being geeky, but then this terminology is cleverly embedded as a syntax in a powerful language. If you are from the Groovy community you must be aware of builders. Groovy builders allow defining data in a semi declarative way. It's a kind of mini-language of its own. Builders are considered good for generating XML and laying out UI components. Kotlin DSL uses Lambdas a lot. DSL in Kotlin are type-safe builders. It means we can detect compilation errors in IntelliJ's beautiful IDE. The type-check builders are much better than the dynamically-typed builders of Groovy. Using Kotlinx.html The DSL to build HTML trees is a pluggable dependency. We, therefore, need to set it up and configure it for our project. For now, we will keep things simple and add the dependency in them in the form of a .jar file. We will keep this .jar file in the lib folder, which will reside at the root level. The library is created by the JetBrains team only and it's open source. You can find it at https://github.com/Kotlin/kotlinx.html. You can simply visit the URL https://dl.bintray.com/kotlin/kotlinx.html/org/jetbrains/kotlinx/kotlinx-html-js/0.6.4/ and download the .jar file from there. For this demo app, we have used v 0.6.4. The .jar repository page can look as follows: To set up the kotlinx.html dependency in your app please follow these steps: In our app, we are using v 0.6.4. Make sure you download the JAR file named  kotlinx-html-js-0.6.4.jar. Please verify that you have kept the .jar file inside the lib directory. Also, do not forget to add the .jar file as a library. Right-click on the .jar file and select Add As Library…. Select classes as a category while adding them as a library. Or you can simply choose to add the dependency via Gradle, in that, you need to add the following things to your build.gradle file: repositories { jcenter() } dependencies { //Fill this in with the version of kotlinx in use in your project def kotlinx_html_version = "your_version_here" // include for client-side compile "org.jetbrains.kotlinx:kotlinx-html- js:${kotlinx_html_version}" } Refactoring the HTML code using DSL The DSL code to make a button with the title "Get Weather" looks as follows: button { +"Get Weather" type = ButtonType.button onClickFunction = { // Your code to handle button click goes here. } } Simple and clean code. Similarly, let's create a function that will display an entire div, which has a label, text input, and button: fun getInputDiv(): HTMLDivElement { val inputDiv = document.create.div { label { +"Enter zip code : " input { id = "zipCode" type = InputType.number value = 411021.toString() } } button { +"Get Weather" type = ButtonType.button onClickFunction = { // Your code to handle button click goes here } } } return inputDiv } Observe how we have provided ID, input types, and a default ZIP code value. A default ZIP code value is optional. Let's spend some time understanding the previous code. label, input, button, type, id, and onClickFunction are nothing but functions. They are basically Lambda functions. Some of the functions that use Lambda parameters and call variations can be as follows: someFunction({}) someFunction("KotlinBluePrints",1,{}) someFunction("KotlinBluePrints",1){} someFunction{} Let's run the code. You may get an error on the console saying: Error Uncaught Error: Error loading module 'KotlinBluePrintsJSDemo'. Its dependency 'kotlinx-html-js' was not found. Please, check whether 'kotlinx-html-js' is loaded prior to 'KotlinBluePrintsJSDemo'. This is because kotlinx-html-js is missing, which is required to process the DSL generated code. You can see the kotlinx-html-js file generated under the out/production/KotlinBluePrintsJSDemo/lib path. Calling a weather API Now it's time to get the weather data and display it on the page. We will use XMLHttpRequest to achieve this. Register yourself at http://openweathermap.org/appid and get your application ID. Your application ID will be appended to the actual URL to make the authenticated call to the weather API. Once you get the app ID let's keep that information in the Constants.kt file: const val IMAGE_URL = "http://openweathermap.org/img/w/%s.png" const val BASE_URL = "https://api.openweathermap.org/data/2.5/forecast/daily? mode=json&units=metric&cnt=7" const val APP_ID = "Your open weather map application id" const val FULL_URL = "$BASE_URL&appid=$APP_ID&q=" The Constants.kt file is not as simple as it looks. Check how we have stored different values. We have used const val, which is equivalent to const and static used combined. Also defining FULL_URL uses the concept of string interpolation. String interpolation is used to concatenate static strings along with string objects. You can also call functions in string interpolation as follows: h4 { +"Weather info for ${forecastResult.city.name}, (${forecastResult.city.country})" } Now, in onClickFunction we write the following code to perform the API call and on the successful response we call a showData function, which takes a forecastResult object: onClickFunction = { val zipCode = document.getElementById("zipCode") as HTMLInputElement val xmlHttpRequest = XMLHttpRequest() xmlHttpRequest.open("GET", FULL_URL + zipCode.value, false) xmlHttpRequest.send() println(xmlHttpRequest.responseText) val forecastResult = JSON.parse<ForecastResult> (xmlHttpRequest.responseText) showData(forecastResult) } Reading data from input elements See how we read data from input elements: document.getElementById("zipCode") as HTMLInputElement The as HTMLInputElement construct is basically casting a result into the HTMLInputElement class. Using as directly is not advisable because it can give you ClassCastException; a proper way to use it is as? HTMLInputElement. This returns null if the class cast fails. And Kotlin will force you to use a Null Safety operator from that very moment. Data classes We are maintaining ForecastResult, which is our model. For this purpose, we have data classes in Kotlin. One of the coolest features in Kotlin is data classes. All the pain that we used to endure to create and maintain POJO classes in Java is gone. No need to have those dedicated packages to hold your model class. Any Kotlin file can hold your data class. By default it provides you methods such as toString(), equals(), copy(), and hashCode() method implementation. In Android, we mostly use these types of classes to hold our JSON responses in the form of model classes. You can check out the data classes we created in ServerResponses.kt: data class ForecastResult(val city: City, val list: Array<Forecast>) data class City(val id: Long, val name: String, val coord: Coordinates, val country: String, val population: Int) data class Coordinates(val lon: Float, val lat: Float) data class Forecast(val dt: Long, val temp: Temperature, val pressure: Float, val humidity: Int, val weather: Array<Weather>, val speed: Float, val deg: Int, val clouds: Int) data class Temperature(val day: Float, val min: Float, val max: Float, val night: Float, val eve: Float, val morn: Float) data class Weather(val id: Long, val main: String, val description: String, val icon: String) Some of the points to consider while using data classes are: The primary constructor needs to have at least one parameter All primary constructor parameters need to be marked as val or var Data classes cannot be abstract, open, sealed, or inner (Before version 1.1) data classes may only implement interfaces Showing data to the user Now comes the interesting part. We gate a ForecastResult object, which holds all the records. The list object holds records for seven days. Let's create a showData function that takes a ForecastResult object and display title text in <h4>.  The code will look like the following snippet. Also, it has yet again one more example of string interpolation: fun showData(forecastResult: ForecastResult) { val root = document.getElementById("container") root?.appendChild(document.create.div(classes = "currentTemp") { h4 { +"Weather info for ${forecastResult.city.name (${forecastResult.city.country})" } }) } This is simple now, quickly create a showForecast function that will be called from showData and will display the weather forecast for seven days. The showForecast is used with a function from Kotlin.  thewith() is one of those functions that is liked by the developer community a lot; it makes use of Kotlin sweeter. The with() function accepts the receiver and the code written inside the function automatically applies to the receiver object. It's an inline function. Check out the following document: /** * Calls the specified function [block] with the given [receiver] as its receiver and returns its result. */ public inline fun <T, R> with(receiver: T, block: T.() -> R): R = receiver.block() In the code, observe how each iteration is using a with block. We have removed some of the lines from the original code, so that we can have the clean code snippet here: forecastResult.list.forEachIndexed { index, forecast -> with(forecast) { weatherContainer.appendChild(document.create.div(classes = "weatherBlock") { div { p(classes = "currentTemp") { +"${Math.round(temp.day)} °C" } } img(classes = "weatherImage") { src = "images/weather_img.png" } div { span(classes = "secondaryText") { +weather[0].main } } div { with(temp) { span(classes = "primaryText") { +"${Math.round(max)} °C" } span(classes = "secondaryText") { +" /${Math.round(min)} °C" } } } onClickFunction = { showDetailedForecast(forecastResult.city, forecast) } }) } } DSL and Kotlin code are now beautifully gelled. Also, notice the onClickFunction that we wrote on div.  Sweet, isn't it? Showing weather details A very small part of the app is left now. Let's show some more details to the user. Along with this, we will also learn a few more features of Kotlin. We have created a showDetailedForecast function that takes the City and Forecast objects as parameters. The following code snippets provide two things to learn: fun showDetailedForecast(city: City, forecast: Forecast) { val root = document.getElementById("container") val weatherDetailDiv = document.create.div(classes = "detailsContainer") val basicDetailDiv = document.create.div { p(classes = "secondaryText") { +"${city.name}, ${city.country} (${city.coord.lat},${city.coord.lon})" } p(classes = "secondaryText") { +forecast.dt.getFullDate() } p(classes = "secondaryText") { +"${forecast.weather[0].main}, ${forecast.weather[0].description}" } } val otherDetailsDiv = document.create.div { div { id = "leftDiv" span(classes = "currentTemp") { +"${Math.round(forecast.temp.day)} °C" } img { src = "images/weather_img.png" width = 90.toString() height = 90.toString() } } div { id = "rightDiv" p(classes = "secondaryText") { +"Pressure: ${forecast.pressure} mb" } p(classes = "secondaryText") { +"Humidity: ${forecast.humidity} %" } p(classes = "secondaryText") { +"Wind: ${forecast.speed} mph" } p(classes = "secondaryText") { +"Cloudiness: ${forecast.clouds} %" } } div(classes = "clearBoth") } weatherDetailDiv.appendChild(basicDetailDiv) weatherDetailDiv.appendChild(otherDetailsDiv) root?.appendChild(weatherDetailDiv) } Named parameters In Kotlin, we can call/bind a parameter with their name for any function. We can call the preceding function by interchanging the parameter sequence as well. Something like the following: showDetailedForecast(forecast = forecast, city = forecastResult.city) Observe that we swapped the place of the variable. And no wonder, all CSS classes that we have applied so far have a named parameter. Check all previous <div>, <h>, and <p> tags. Consider the following examples: val weatherDetailDiv = document.create.div(classes = "detailsContainer") button(classes = "getWeatherButton") span(classes = "primaryText") { +"${Math.round(max)} °C" } Extension functions Extension functions are a beautiful feature of Kotlin. Extension functions allow us to add the functions in the native class sets. All extension functions are statically resolved. Check out DateExtension.kt, it has three extension functions written for Long objects. They return different date formats. The code inside it may look a bit strange, which we will discuss in the following section: fun Long.getShortDate(): String { val getFormattedDate: dynamic = js("window.getShortDate") return getFormattedDate(this) } fun Long.getFullDate(): String { val getFormattedDate: dynamic = js("window.getFullDate") return getFormattedDate(this) } fun Long.getFullWeekDay(): String { val getFormattedDate: dynamic = js("window.getFullWeekDay") return getFormattedDate(this) } We don't need to write utility methods in Kotlin. We should prefer extension functions over Utils. Do not try to have any heavy methods as extension functions, instance functions are always good. Writing extension functions to format dates and to have some validation functions is OK. But it's not good to write an API calling function for any string class. Remember they are statically resolved. A project loaded with static is not good for memory. Giving final touches We wrote many lines of code so far. We also refactored them periodically. Once again it's a time to refactor and look for the possible improvements. Let's take a look back and see if there is any possibility of refactoring the code further. Adding CSS Let's add some custom font and style some of the missed HTML elements. We have used Robot font, you can use any font of your desire. It's a simple one-liner code to mention the font in the app. Add the following line to your index.html page just after the <body> tag: <link href="https://fonts.googleapis.com/css? family=Roboto+Condensed" rel="stylesheet"> And in main.css apply the font to an entire HTML page: html * { font-family: 'Roboto Condensed', sans-serif; } Reload the page. Looks beautiful now, doesn't it? To summarize, we learned various elements of Kotlin such as setting up Kotlin for JavaScript projects, interacting with DOM elements, DSL, and so on. The purpose of this article was to show that Kotlin's support for JavaScript is no more an experiment. It's already production ready. You can see what can be done using the benefits of statically typed programming languages and powerful JavaScript ecosystems. To know more about how to use Kotlin code for writing a Node.js application, you may refer to this book Kotlin Blueprints. Build your first Android app with Kotlin How to convert Java code into Kotlin 5 application development tools that will matter in 2018  

Ext JS 4 is Sencha’s latest JavaScript framework for developing cross-platform web applications. Built upon web standards, Ext JS provides a comprehensive library of user interface widgets and data manipulation classes to turbo-charge your application’s development. In this article, written by Stuart Ashworth and Andrew Duncan, the authors of Ext JS 4 Web Application Development Cookbook, we will cover: Constructing a complex form layout Populating your form with data Submitting your form's data Validating form fields with VTypes Creating custom VTypes Uploading files to the server Handling exceptions and callbacks This article introduces forms in Ext JS 4. We begin by creating a support ticket form in the first recipe. To get the most out of this article you should be aware that this form is used by a number of recipes throughout the article. Instead of focusing on how to configure specific fields, we demonstrate more generic tasks for working with forms. Specifically, these are populating forms, submitting forms, performing client-side validation, and handling callbacks/exceptions. Constructing a complex form layout In the previous releases of Ext JS, complicated form layouts were quite difficult to achieve. This was due to the nature of the FormLayout, which was required to display labels and error messages correctly, and how it had to be combined with other nested layouts. Ext JS 4 takes a different approach and utilizes the Ext.form.Labelable mixin, which allows form fields to be decorated with labels and error messages without requiring a specific layout to be applied to the container. This means we can combine all of the layout types the framework has to offer without having to overnest components in order to satisfy the form field's layout requirements. We will describe how to create a complex form using multiple nested layouts and demonstrate how easy it is to get a form to look exactly as we want. Our example will take the structure of a Support Ticket Request form and, once we are finished, it will look like the following screenshot: (Move the mouse over the image to enlarge.) How to do it... We start this recipe by creating a simple form panel that will contain all of the layout containers and their fields: var formPanel = Ext.create('Ext.form.Panel', { title: 'Support Ticket Request', width: 650, height: 500, renderTo: Ext.getBody(), style: 'margin: 50px', items: [] }); Now, we will create our first set of fields— the FirstName and LastName fields. These will be wrapped in an Ext.container.Container component, which is given an hbox layout so our fields appear next to each other on one line: var formPanel = Ext.create('Ext.form.Panel', { title: 'Support Ticket Request', width: 650, height: 500, renderTo: Ext.getBody(), style: 'margin: 50px', items: [{ xtype: 'container', layout: 'hbox', items: [{ xtype: 'textfield', fieldLabel: 'First Name', name: 'FirstName', labelAlign: 'top', cls: 'field-margin', flex: 1 }, { xtype: 'textfield', fieldLabel: 'Last Name', name: 'LastName', labelAlign: 'top', cls: 'field-margin', flex: 1 }] }] }); We have added a CSS class (field-margin) to each field, to provide some spacing between them. We can now add this style inside <style> tags in the head of our document: <style type="text/css"> .field-margin { margin: 10px; }</style> Next, we create a container with a column layout to position our e-mail address and telephone number fields. We nest our telephone number fields in an Ext.form.FieldContainer class , which we will discuss later in the recipe: items: [ ... { xtype: 'container', layout: 'column', items: [{ xtype: 'textfield', fieldLabel: 'Email Address', name: 'EmailAddress', labelAlign: 'top', cls: 'field-margin', columnWidth: 0.6 }, { xtype: 'fieldcontainer', layout: 'hbox', fieldLabel: 'Tel. Number', labelAlign: 'top', cls: 'field-margin', columnWidth: 0.4, items: [{ xtype: 'textfield', name: 'TelNumberCode', style: 'margin-right: 5px;', flex: 2 }, { xtype: 'textfield', name: 'TelNumber', flex: 4 }] }] } ... ] The text area and checkbox group are created and laid out in a similar way to the previous sets, by using an hbox layout: items: [ ... { xtype: 'container', layout: 'hbox', items: [{ xtype: 'textarea', fieldLabel: 'Request Details', name: 'RequestDetails', labelAlign: 'top', cls: 'field-margin', height: 250, flex: 2 }, { xtype: 'checkboxgroup', name: 'RequestType', fieldLabel: 'Request Type', labelAlign: 'top', columns: 1, cls: 'field-margin', vertical: true, items: [{ boxLabel: 'Type 1', name: 'type1', inputValue: '1' }, { boxLabel: 'Type 2', name: 'type2', inputValue: '2' }, { boxLabel: 'Type 3', name: 'type3', inputValue: '3' }, { boxLabel: 'Type 4', name: 'type4', inputValue: '4' }, { boxLabel: 'Type 5', name: 'type5', inputValue: '5' }, { boxLabel: 'Type 6', name: 'type6', inputValue: '6' }], flex: 1 }] } ... ] Finally, we add the last field, which is a file upload field, to allow users to provide attachments: items: [ ... { xtype: 'filefield', cls: 'field-margin', fieldLabel: 'Attachment', width: 300 } ... ] How it works... All Ext JS form fields inherit from the base Ext.Component class and so can be included in all of the framework's layouts. For this reason, we can include form fields as children of containers with layouts (such as hbox and column layouts) and their position and size will be calculated accordingly. Upgrade Tip: Ext JS 4 does not have a form layout meaning a level of nesting can be removed and the form fields' labels will still be displayed correctly by just specifying the fieldLabel config. The Ext.form.FieldContainer class used in step 4 is a special component that allows us to combine multiple fields into a single container, which also implements the Ext.form. Labelable mixin . This allows the container itself to display its own label that applies to all of its child fields while also giving us the opportunity to configure a layout for its child components. Populating your form with data After creating our beautifully crafted and user-friendly form we will inevitably need to populate it with some data so users can edit it. Ext JS makes this easy, and this recipe will demonstrate four simple ways of achieving it. We will start by explaining how to populate the form on a field-by-field basis, then move on to ways of populating the entire form at once. We will also cover populating it from a simple object, a Model instance, and a remote server call. Getting ready We will be using the form created in this article's first recipe as our base for this section, and many of the subsequent recipes in this article, so please look back if you are not familiar with it. All the code we will write in this recipe should be placed under the definition of this form panel. You will also require a working web server for the There's More example, which loads data from an external file. How to do it... We'll demonstrate how to populate an entire form's fields in bulk and also how to populate them individually. Populating individual fields We will start by grabbing a reference to the first name field using the items property's get method. The items property contains an instance of Ext.util. MixedCollection, which holds a reference to each of the container's child components. We use its get method to retrieve the component at the specified index: var firstNameField = formPanel.items.get(0).items.get(0); Next, we use the setValue method of the field to populate it: firstNameField.setValue('Joe'); Populating the entire form To populate the entire form, we must create a data object containing a value for each field. The property names of this object will be mapped to the corresponding form field by the field's name property. For example, the FirstName property of our requestData object will be mapped to a form field with a name property value of FirstName: var requestData = { FirstName: 'Joe', LastName: 'Bloggs', EmailAddress: '[email protected]', TelNumberCode: '0777', TelNumber: '7777777', RequestDetails: 'This is some Request Detail body text', RequestType: { type1: true, type2: false, type3: false, type4: true, type5: true, type6: false } }; We then call the setValues method of the form panel's Ext.form.Basic instance, accessed through the getForm method, passing it our requestData variable: formPanel.getForm().setValues(requestData); How it works... Each field contains a method called setValue , which updates the field's value with the value that is passed in. We can see this in action in the first part of the How to do it section. A form panel contains an internal instance of the Ext.form.Basic class (accessible through the getForm method ), which provides all of the validation, submission, loading, and general field management that is required by a form. This class contains a setValues method , which can be used to populate all of the fields that are managed by the basic form class. This method works by simply iterating through all of the fields it contains and calling their respective setValue methods. This method accepts either a simple data object, as in our example, whose properties are mapped to fields based on the field's name property. Alternatively, an array of objects can be supplied, containing id and value properties, with the id mapping to the field's name property. The following code snippet demonstrates this usage: formPanel.getForm().setValues([{id: 'FirstName', value: 'Joe'}]);   There's more... Further to the two previously discussed methods there are two others that we will demonstrate here. Populating a form from a Model instance Being able to populate a form directly from a Model instance is extremely useful and is very simple to achieve. This allows us to easily translate our data structures into a form without having to manually map it to each field. We initially define a Model and create an instance of it (using the data object we used earlier in the recipe): Ext.define('Request', { extend: 'Ext.data.Model', fields: [ 'FirstName', 'LastName', 'EmailAddress', 'TelNumberCode', 'TelNumber', 'RequestDetails', 'RequestType' ] }); var requestModel = Ext.create('Request', requestData); Following this we call the loadRecord method of the Ext.form.Basic class and supply the Model instance as its only parameter. This will populate the form, mapping each Model field to its corresponding form field based on the name: formPanel.getForm().loadRecord(requestModel); Populating a form directly from the server It is also possible to load a form's data directly from the server through an AJAX call. Firstly, we define a JSON file, containing our request data, which will be loaded by the form: { "success": true, "data": { "FirstName": "Joe", "LastName": "Bloggs", "EmailAddress": "[email protected]", "TelNumberCode": "0777", "TelNumber": "7777777", "RequestDetails": "This is some Request Detail body text", "RequestType": { "type1": true, "type2": false, "type3": false, "type4": true, "type5": true, "type6": false } } } Notice the format of the data: we must provide a success property to indicate that the load was successful and put our form data inside a data property. Next we use the basic form's load method and provide it with a configuration object containing a url property pointing to our JSON file: formPanel.getForm().load({ url: 'requestDetails.json' }); This method automatically performs an AJAX request to the specified URL and populates the form's fields with the data that was retrieved. This is all that is required to successfully load the JSON data into the form. The basic form's load method accepts similar configuration options to a regular AJAX request Submitting your form's data Having taken care of populating the form it's now time to look at sending newly added or edited data back to the server. As with form population you'll learn just how easy this is with the Ext JS framework. There are two parts to this example. Firstly, we will submit data using the options of the basic form that wraps the form panel. The second example will demonstrate binding the form to a Model and saving our data. Getting ready We will be using the form created in the first recipe as our base for this section, so refer to the Constructing a complex form layout recipe, if you are not familiar with it. How to do it... Add a function to submit the form: var submitForm = function(){ formPanel.getForm().submit({ url: 'submit.php' }); }; Add a button to the form that calls the submitForm function: var formPanel = Ext.create('Ext.form.Panel', { ... buttons: [{ text: 'Submit Form', handler: submitForm }], items: [ ... ] }); How it works... As we learned in the previous recipe, a form panel contains an internal instance of the Ext.form.Basic class (accessible through the getForm method). The submit method in Ext.form.Basic is a shortcut to the Ext.form.action.Submit action. This class handles the form submission for us. All we are required to do is provide it with a URL and it will handle the rest. It's also possible to define the URL in the configuration for the Ext.form.Panel.. Before submitting, it must first gather the data from the form. The Ext.form.Basic class contains a getValues method, which is used to gather the data values for each form field. It does this by iterating through all fields in the form making a call to their respective getValue methods. There's more... The previous recipe demonstrated how to populate the form from a Model instance. Here we will take it a step further and use the same Model instance to submit the form as well. Submitting a form from a Model instance Extend the Model with a proxy and load the data into the form: xt.define('Request', { extend: 'Ext.data.Model', fields: ['FirstName', 'LastName', 'EmailAddress', 'TelNumberCode', 'TelNumber', 'RequestDetails', 'RequestType'], proxy: { type: 'ajax', api: { create: 'addTicketRequest.php', update: 'updateTicketRequest.php' }, reader: { type: 'json' } } }); var requestModel = Ext.create('Request', { FirstName: 'Joe', LastName: 'Bloggs', EmailAddress: '[email protected]' }); formPanel.getForm().loadRecord(requestModel); Change the submitForm function to get the Model instance, update the record with the form data, and save the record to the server: var submitForm = function(){ var record = formPanel.getForm().getRecord(); formPanel.getForm().updateRecord(record); record.save(); }; Validating form fields with VTypes In addition to form fields' built-in validation (such as allowBlank and minLength), we can apply more advanced and more extensible validation by using VTypes. A VType (contained in the Ext.form.field.VTypes singleton) can be applied to a field and its validation logic will be executed as part of the field's periodic validation routine. A VType encapsulates a validation function, an error message (which will be displayed if the validation fails), and a regular expression mask to prevent any undesired characters from being entered into the field. This recipe will explain how to apply a VType to the e-mail address field in our example form, so that only properly formatted e-mail addresses are deemed valid and an error will be displayed if it doesn't conform to this pattern. How to do it... We will start by defining our form and its fields. We will be using our example form that was created in the first recipe of this article as our base. Now that we have a form we can add the vtype configuration option to our e-mail address field: { xtype: 'textfield', fieldLabel: 'Email Address', name: 'EmailAddress', labelAlign: 'top', cls: 'field-margin', columnWidth: 0.6, vtype: 'email' } That is all we have to do to add e-mail address validation to a field. We can see the results in the following screenshot, with an incorrectly formatted e-mail address on the left and a valid one on the right: How it works... When a field is validated it runs through various checks. When a VType is defined the associated validation routine is executed and will flag the field invalid or not . As previously mentioned, each VType has an error message coupled with it, which is displayed if it is found to be invalid, and a mask expression which prevents unwanted characters being entered. Unfortunately, only one VType can be applied to a field and so, if multiple checks are required, a custom hybrid may need to be created. See the next recipe for details on how to do this. There's more... Along with the e-mail VType, the framework provides three other VTypes that can be applied straight out of the box. These are: alpha: this restricts the field to only alphabetic characters alphnum: this VType allows only alphanumeric characters url: this ensures that the value is a valid URL Creating custom VTypes We have seen in the previous recipe how to use VTypes to apply more advanced validation to our form's fields. The built-in VTypes provided by the framework are excellent but we will often want to create custom implementations to impose more complex and domain specific validation to a field. We will walkthrough creating a custom VType to be applied to our telephone number field to ensure it is in the format that a telephone number should be. Although our telephone number field is split into two (the first field for the area code and the second for the rest of the number), for this example we will combine them so our VType is more comprehensive. For this example, we will be validating a very simple, strict telephone number format of "0777-777-7777". How to do it... We start by defining our VType's structure. This consists of a simple object literal with three properties. A function called telNumber and two strings called telNumberText (which will contain the error message text) and telNumberMask (which holds a regex to restrict the characters allowed to be entered into the field) respectively. var telNumberVType = { telNumber: function(val, field){ // function executed when field is validated // return true when field's value (val) is valid return true; }, telNumberText: 'Your Telephone Number must only include numbers and hyphens.', telNumberMask: /[d-]/ }; Next we define the regular expression that we will use to validate the field's value. We add this as a variable to the telNumber function: telNumber: function(val, field){ var telNumberRegex = /^d{4}-d{3}-d{4}$/; return true; } Once this has been done we can add the logic to this telNumber function that will decide whether the field's current value is valid. This is a simple call to the regular expression string's test method, which returns true if the value matches or false if it doesn't: telNumber: function(val, field){ var telNumberRegex = /^d{4}-d{3}-d{4}$/; return telNumberRegex.test(val); } The final step to defining our new VType is to apply it to the Ext.form.field. VTypes singleton, which is where all of the VTypes are located and where our field's validation routine will go to get its definition: Ext.apply(Ext.form.field.VTypes, telNumberVType); Now that our VType has been defined and registered with the framework, we can apply it to the field by using the vtype configuration option. The result can be seen in the following screenshot: { xtype: 'textfield', name: 'TelNumber', flex: 4, vtype: 'telNumber' } How it works... A VType consists of three parts: The validity checking function The validation error text A keystroke filtering mask (optional) VTypes rely heavily on naming conventions so they can be executed dynamically within a field's validation routine. This means that each of these three parts must follow the standard convention. The validation function's name will become the name used to reference the VType and form the prefix for the other two properties. In our example, this name was telNumber, which can be seen referencing the VType in Step 5. The error text property is then named with the VType's name prefixing the word Text (that is, telNumberText ). Similarly, the filtering mask is the VType's name followed by the word Mask (that is, telNumberMask ). The final step to create our VType is to merge it into the Ext.form.field.VTypes singleton allowing it to be accessed dynamically during validation. The Ext.apply function does this by merging the VType's three properties into the Ext.form.field.VTypes class instance. When the field is validated, and a vtype is defined, the VType's validation function is executed with the current value of the field and a reference to the field itself being passed in. If the function returns true then all is well and the routine moves on. However, if it evaluates to false the VType's Text property is retrieved and pushed onto the errors array. This message is then displayed to the user as our screenshot shown earlier. This process can be seen in the code snippet as follows, taken directly from the framework: if (vtype) { if(!vtypes[vtype](value, me)){ errors.push(me.vtypeText || vtypes[vtype +'Text']); } } There's more... It is often necessary to validate fields based on the values of other fields as well as their own. We will demonstrate this by creating a simple VType for validating that a confirm password field's value matches the value entered in an initial password field. We start by creating our VType structure as we did before: Ext.apply(Ext.form.field.VTypes, { password: function(val, field){ return false; }, passwordText: 'Your Passwords do not match.' }); We then complete the validation logic. We use the field's up method to get a reference to its parent form. Using that reference, we get the values for all of the form's fields by using the getValues method : password: function(val, field){ var parentForm = field.up('form'); // get parent form // get the form's values var formValues = parentForm.getValues(); return false; } The next step is to get the first password field's value. We do this by using an extra property ( firstPasswordFieldName) that we will specify when we add our VType to the confirm password field. This property will contain the name of the initial password field (in this example Password ). We can then compare the confirm password's value with the retrieved value and return the outcome: password: function(val, field){ var parentForm = field.up('form'); // get parent form // get the form's values var formValues = parentForm.getValues(); // get the value from the configured 'First Password' field var firstPasswordValue = formValues[field.firstPasswordFieldName]; // return true if they match return val === firstPasswordValue; } The VType is added to the confirm password field in exactly the same way as before but we must include the extra firstPasswordFieldName option to link the fields together: { xtype: 'textfield', fieldLabel: 'Confirm Password', name: 'ConfirmPassword', labelAlign: 'top', cls: 'field-margin', flex: 1, vtype: 'password', firstPasswordFieldName: 'Password' } Uploading files to the server Uploading files is very straightforward with Ext JS 4. This recipe will demonstrate how to create a basic file upload form and send the data to your server: Getting Ready This recipe requires the use of a web server for accepting the uploaded file. A PHP file is provided to handle the file upload; however, you can integrate this Ext JS code with any server-side technology you wish. How to do it... Create a simple form panel. Ext.create('Ext.form.Panel', { title: 'Document Upload', width: 400, bodyPadding: 10, renderTo: Ext.getBody(), style: 'margin: 50px', items: [], buttons: [] }); In the panel's items collection add a file field: Ext.create('Ext.form.Panel', { ... items: [{ xtype: 'filefield', name: 'document', fieldLabel: 'Document', msgTarget: 'side', allowBlank: false, anchor: '100%' }], buttons: [] }); Add a button to the panel's buttons collection to handle the form submission: Ext.create('Ext.form.Panel', { ... buttons: [{ text: 'Upload Document', handler: function(){ var form = this.up('form').getForm(); if (form.isValid()) { form.submit({ url: 'upload.php', waitMsg: 'Uploading...' }); } } }] }); How it works... Your server-side code should handle these form submissions in the same way they would handle a regular HTML file upload form. You should not have to do anything special to make your server-side code compatible with Ext JS. The example works by defining an Ext.form.field.File ( xtype: 'filefield' ), which takes care of the styling and the button for selecting local files. The form submission handler works the same way as any other form submission; however, behind the scenes the framework tweaks how the form is submitted to the server. A form with a file upload field is not submitted using an XMLHttpRequest object—instead the framework creates and submits a temporary hidden <form> element whose target is referenced to a temporary hidden <iframe>. The request header's Content-Type is set to multipart/form. When the upload is finished and the server has responded, the temporary form and <iframe> are removed. A fake XMLHttpRequest object is then created containing a responseText property (populated from the contents of the <iframe> ) to ensure that event handlers and callbacks work as if we were submitting the form using AJAX. If your server is responding to the client with JSON, you must ensure that the response Content-Type header is text/html. There's more... It's possible to customize your Ext.form.field.File. Some useful config options are highlighted as follows: buttonOnly: Boolean Setting buttonOnly: true removes the visible text field from the file field. buttonText: String If you wish to change the text in the button from the default of "Browse…" it's possible to do so by setting the buttonText config option. buttonConfig: Object Changing the entire configuration of the button is done by defining a standard Ext.button. Button config object in the buttonConfig option. Anything defined in the buttonText config option will be ignored if you use this. Handling exception and callbacks This recipe demonstrates how to handle callbacks when loading and submitting forms. This is particularly useful for two reasons: You may wish to carry our further processing once the form has been submitted (for example, display a thank you message to the user) In the unfortunate event when the submission fails, it's good to be ready and inform the user something has gone wrong and perhaps perform extra processing The recipe shows you what to do in the following circumstances: The server responds informing you the submission was successful The server responds with an unusual status code (for example, 404 , 500 , and so on) The server responds informing you the submission was unsuccessful (for example, there was a problem processing the data) The form is unable to load data because the server has sent an empty data property The form is unable to submit data because the framework has deemed the values in the form to be invalid Getting ready The following recipe requires you to submit values to a server. An example submit.php file has been provided. However, please ensure you have a web server for serving this file. How to do it... Start by creating a simple form panel: var formPanel = Ext.create('Ext.form.Panel', { title: 'Form', width: 300, bodyPadding: 10, renderTo: Ext.getBody(), style: 'margin: 50px', items: [], buttons: [] }); Add a field to the form and set allowBlank to false: var formPanel = Ext.create('Ext.form.Panel', { ... items: [{ xtype: 'textfield', fieldLabel: 'Text field', name: 'field', allowBlank: false }], buttons: [] }); Add a button to handle the forms submission and add success and failure handlers to the submit method's only parameter: var formPanel = Ext.create('Ext.form.Panel', { ... buttons: [{ text: 'Submit', handler: function(){ formPanel.getForm().submit({ url: 'submit.php', success: function(form, action){ Ext.Msg.alert('Success', action.result.message); }, failure: function(form, action){ if (action.failureType === Ext.form.action.Action. CLIENT_INVALID) { Ext.Msg.alert('CLIENT_INVALID', 'Something has been missed. Please check and try again.'); } if (action.failureType === Ext.form.action.Action. CONNECT_FAILURE) { Ext.Msg.alert('CONNECT_FAILURE', 'Status: ' + action.response.status + ': ' + action.response.statusText); } if (action.failureType === Ext.form.action.Action. SERVER_INVALID) { Ext.Msg.alert('SERVER_INVALID', action.result. message); } } }); } }] }); When you run the code, watch for the different failureTypes or the success callback: CLIENT_INVALID is fired when there is no value in the text field. The success callback is fired when the server returns true in the success property. Switch the response in submit.php file and watch for SERVER_INVALID failureType. This is fired when the success property is set to false. Finally, edit url: 'submit.php' to url: 'unknown.php' and CONNECT_FAILURE will be fired. How it works... The Ext.form.action.Submit and Ext.form.action.Load classes both have a failure and success function. One of these two functions will be called depending on the outcome of the action. The success callback is called when the action is successful and the success property is true. The failure callback , on the other hand, can be extended to look for specific reasons why the failure occurred (for example, there was an internal server error, the form did not pass client-side validation, and so on). This is done by looking at the failureType property of the action parameter. Ext.form.action.Action has four failureType static properties: CLIENT_INVALID, SERVER_INVALID, CONNECT_FAILURE, and LOAD_FAILURE, which can be used to compare with what has been returned by the server. There's more... A number of additional options are described as follows: Handling form population failures The Ext.form.action.Action.LOAD_FAILURE static property can be used in the failure callback when loading data into your form. The LOAD_FAILURE is returned as the action parameter's failureType when the success property is false or the data property contains no fields. The following code shows how this failure type can be caught inside the failure callback function: failure: function(form, action){ ... if(action.failureType == Ext.form.action.Action.LOAD_FAILURE){ Ext.Msg.alert('LOAD_FAILURE', action.result.message); } ... } An alternative to CLIENT_INVALID The isValid method in Ext.form.Basic is an alternative method for handling client-side validation before the form is submitted. isValid will return true when client-side validation passes: handler: function(){ if (formPanel.getForm().isValid()) { formPanel.getForm().submit({ url: 'submit.php' }); } }   Further resources on this subject: Ext JS 4: Working with the Grid Component [Article] Ext JS 4: Working with Tree and Form Components [Article] Infinispan Data Grid: Infinispan and JBoss AS 7 [Article]

In the Article by Gerardo Barajas Puente, author of Elastix Unified Communications Server Cookbook, we will discuss some topics regarding security in our Elastix Unified Communications System. We will share some recommendations to ensure our system's availability, privacy, and correct performance. Attackers' objectives may vary from damaging data, to data stealing, to telephonic fraud, to denial of service. This list is intended to minimize any type of attack, but remember that there are no definitive arguments about security; it is a constantly changing subject with new types of attacks, challenges, and opportunities. (For more resources related to this topic, see here.) The recipes covered in this article are as follows: Using Elastix's embedded firewall Using the Security Advanced Settings menu to enable security features Recording and monitoring calls Recording MeetMe rooms (conference rooms) Recording queues' calls Monitoring recordings Upgrading our Elastix system Generating system backups Restoring a backup from one server to another Using Elastix's embedded firewall Iptables is one of the most powerful tools of Linux's kernel. It is used largely in servers and devices worldwide. Elastix's security module incorporates iptables' main features into its webGUI in order to secure our Unified Communications Server. This module is available in the Security | Firewall menu. In this module's main screen, we can check the status of the firewall (Activated or Deactivated). We will also notice the status of each rule of the firewall with the following information: Order: This column represents the order in which rules will be applied Traffic: The rule will be applied to any ingoing or outgoing packet Target: This option allows, rejects, or drops a packet Interface: This represents the network interface on which the rule will be used Source Address: The firewall will search for this IP source address and apply the rule. Destination Address: We can apply a firewall rule if the destination address is matched Protocol: We can apply a rule depending on the IP protocol of the packet (TCP, UDP, ICMP, and so on) Details: In this column, the details or comments regarding this rule may appear in order to remind us of why this rule is being applied By default, when the firewall is applied, Elastix will allow the traffic from any device to use the ports that belong to the Unified Communications Services. The next image shows the state of the firewall. We can review this information in the Define Ports section as shown in the next image: In this section, we can delete, define a new rule (or port), or search for a specific port. If we click on the View link, we will be redirected to the editing page for the selected rule as shown in the next picture. This is helpful whenever we would like to change the details of a rule. How to do it… To add a new rule, click on the Define Port link and add the following information as shown in the next image:     Name: Name for this port.     Protocol: We can choose the IP protocol to use. The options are as follows: TCP, ICMP, IP, and UDP.     Port: We can enter a single port or a range of ports. To enter a port we just enter the port number in the first text field before the ":" character. If we'd like to enter a range, we must use the two text areas. The first one is for the first port of the range, and the second one is for the last port of the range.     Comment: We can enter a comment for this port. The next image shows the creation of a new port for GSM-Solution. This solution will use the TCP protocol from port 5000 to 5002. Having our ports defined, we proceed to activate the firewall by clicking on Save. As soon as the firewall service is activated, we will see the status of every rule. A message will be displayed, informing us that the service has been activated. When the service has been started, we will be able to edit, eliminate or change the execution order of a certain rule or rules. To add a new rule, click on the New Rule button (as shown in the next picture) and we will be redirected to a new web page. The information we need to enter is as follows:     Traffic: This option sets the rule for incoming (INPUT), outgoing (OUTPUT), or redirecting (FORWARD) packets.     Interface IN: This is the interface used for the rule. All the available network interfaces will be listed. The options ANY and LOOPBACK are also available     Source Address: We can apply a rule for any specified IP address. For example, we can block all the incoming traffic from the IP address 192.168.1.1. It is important to specify its netmask.     Destination Address: This is the destination IP address for the rule. It is important to specify its netmask.     Protocol: We can choose the protocol we would like to filter or forward. The options are TCP, UDP, ICMP, IP, and STATE.     Source Port: In this section, we can choose any option previously configured in the Port Definition section for the source port.     Destination Port: Here, we can select any option previously configured in the Port Definition section for the source port.     Target: This is the action to perform for any packet that matches any of the conditions set in the previous fields The next image shows the application of a new firewall's rule based on the ports we defined previously: We can also check the user's activity by using the Audit menu. This module can be found in the Security menu. To enhance our system's security we also recommend using Elastix's internal Port Knocking feature. Using the Security Advanced Settings menu to enable security features The Advanced Settings option will allow us to perform the following actions: Enable or disable direct access to FreePBX's webGUI. Enable or disable anonymous SIP calls. Change the database and web administration password for FreePBX. How to do it… Click on the Security | Advanced Settings menu and these options are shown as in the next screenshot. Recording and monitoring calls Whenever we have the need for recording the calls that pass through our system, Elastix, and taking advantage of FreePBX's and Asterisk's features. In this section, we will show the configuration steps to record the following types of calls: Extension's inbound and outbound calls MeetMe rooms (conference rooms) Queues Getting ready... Go to PBX | PBX Configuration | General Settings. In the section called Dialing Options, add the values w and W to the Asterisk Dial command options and the Asterisk Outbound Dial command options. These values will allow the users to start recording after pressing *1. The next screenshot shows this configuration. The next step is to set the options from the Call Recording section as follows: Extension recording override: Disabled. If enabled, this option will ignore all automatic recording settings for all extensions. Call recording format: We can choose the audio format that the recording files will have. We recommend the wav49 format because it is compact and the voice is understandable despite the audio quality. Here is a brief description for the audio file format: WAV: This is the most popular good quality recording format, but its size will increase by 1 MB per minute. WAV49: This format results from a GSM codec recording under the WAV encapsulation making the recording file smaller: 100 KB per minute. Its quality is similar to that of a mobile phone call. ULAW/ALAW: This is the native codec (G.711) used between TELCOS and users, but the file size is very large (1 MB per minute). SLN: SLN means SLINEAR format, which is Asterisk's native format. It is an 8-kHz, 16-bit signer linear raw format. GSM: This format is used for recording calls by using the GSM codec. The recording file size will be increased at a rate of 100 KB per minute. Recording location: We leave this option blank. This option specifies the folder where our recordings will be stored. By default, our system is configured to record calls in the /var/spool/asterisk/monitor folder. Run after record: We also leave this option blank. This is for running a script after a recording has been done. For more information about audio formats, visit: http://www.voip-info.org/wiki/view/Convert+WAV+audio+files+for+use+in+Asterisk Apply the changes. All these options are shown in the next screenshot: How to do it… To record all the calls that are generated or received from or to extensions go to the extension's details in the module: PBX | PBX Configuration. We have to click on the desired extension we would like to activate its call recording. In the Recording Options section, we have two options:     Record Incoming     Record Outgoing Depending on the type of recording, select from one of the following options:     On Demand: In this option, the user must press *1 during a call to start recording it. This option only lasts for the current call. When this call is terminated, if the user wants to record another, the digits *1 must be pressed again. If *1 is pressed during a call that is being recorded, the recording will be stopped.     Always: All the calls will be recorded automatically.     Never: This option disables all call recording. These options are shown in the next image. Recording MeetMe rooms If we need to record the calls that go to a conference room, Elastix allows us to do this. This feature is very helpful whenever we need to remember the topics discussed in a conference. How to do it… To record the calls of a conference room, enable it at the conference's details. These details are found in the menu: PBX | PBX Configuration | Conferences. Click on the conference we would like to record and set the Record Conference option to Yes. Save and apply the changes. These steps are shown in the next image. Recording queues' calls Most of the time, the calls that arrive in a queue must be recorded for quality and security purposes. In this recipe, we will show how to enable this feature. How to do it… Go to PBX | PBX Configuration | Queues. Click on a queue to record its calls. Search for the Call Recording option. Select the recording format to use (wav49, wav, gsm). Save and apply the changes. The following image shows the configuration of this feature. Monitoring recordings Now that we know how to record calls, we will show how to retrieve them in order to listen them. How to do it… To visualize the recorded calls, go to PBX | Monitoring. In this menu, we will be able to see the recordings stored in our system. The displayed columns are as follows:     Date: Date of call     Time: Time of call     Source: Source of call (may be an internal or external number)     Destination: Destination of call (may be an internal or external number)     Duration: Duration of call     Type: Incoming or outgoing     Message: This column sets the Listen and Download links to enable you to listen or download the recording files. To listen to a recording, just click on the Message link and a new window will popup in your web browser. This window will have the options to playback the selected recording. It is important to enable our web browser to reproduce audio. To download a recording, we click on the Download link. To delete a recording or group of recordings, just select them and click on the Delete button. To search for a recording or set of recordings, we can do it by date, source, destination, or type, by clicking on the Show Filter button. If click on the Download button, we can download the search or report of the recording files in any of the following formats: CSV, Excel, or Text. It is very important to regularly check the Hard Disk status to prevent it from getting full of recording files and therefore have insufficient space to allow the main services work efficiently. Encrypting voice calls In Elastix/Asterisk, the SIP calls can be encrypted in two ways: encrypting the SIP protocol signaling and encrypting the RTP voice flow. To encrypt the SIP protocol signal, we will use the Transport Layer Security (TLS) protocol. How to do it… Create security keys and certificates. For this example, we will store our keys and certificates in the /etc/asterisk/keys folder. To create this folder, enter the mkdir /etc/asterisk/keys command. Change the owner of the folder from the user root to the user asterisk: chown asterisk:asterisk /etc/asterisk/keys Generate the keys and certificates by going to the following folder: cd /usr/share/doc/asterisk-1.8.20.0/contrib/scripts/   ./ast_tls_cert -C 10.20.30.70 -O "Our Company" -d /etc/asterisk/keys Where the options are as follows:     -C is used to set the host (DNS name) or IP address of our Elastix server.     -O is the organizational name or description.     -d is the folder where keys will be stored. Generate a pair of keys for a pair of extensions (extension 7002 and extension 7003, for example):     For extension 7002: ./ast_tls_cert -m client -c /etc/asterisk/keys/ca.crt -k /etc/asterisk/keys/ca.key -C 10.20.31.107 -O "Elastix Company" -d /etc/asterisk/keys -o 7002     And for extension 7003 ./ast_tls_cert -m client -c /etc/asterisk/keys/ca.crt -k /etc/asterisk/keys/ca.key -C 10.20.31.106 -O "Elastix Company" -d /etc/asterisk/keys -o 7003 where:     -m client: This option sets the program to create a client certificate.     -c /etc/asterisk/keys/ca.crt: This option specifies the Certificate Authority to use (our IP-PBX).     -k /etc/asterisk/keys/ca.key: Provides the key file to the *.crt file.     -C: This option defines the hostname or IP address of our SIP device.     -O: This option defines the organizational name (same as above).     -d: This option specifies the directory where the keys and certificates will be stored.     -o: This is the name of the key and certificate we are creating. When creating the client's keys and certificates, we must enter the same password set when creating the server's certificates. Configure the IP-PBX to support TLS by editing the sip_general_custom.conf file located in the /etc/asterisk/ folder. Add the following lines: tlsenable=yes tlsbindaddr=0.0.0.0 tlscertfile=/etc/asterisk/keys/asterisk.pem tlscafile=/etc/asterisk/keys/ca.crt tlscipher=ALL tlsclientmethod=tlsv1 tlsdontverifyserver=yes     These lines are in charge of enabling the TLS support in our IP-PBX. They also specify the folder where the certificates and the keys are stored and set the ciphering option and client method to use. Add the line transport=tls to the extensions we would like to use TLS in the sip_custom.conf file located at /etc/asterisk/. This file should look like: [7002](+) encryption=yes transport=tls   [7003](+) encryption=yes transport=tls Reload the SIP module in the Asterisk service. This can be done by using the command: asterisk -rx 'sip reload' Configure our TLS-supporting IP phones. This configuration varies from model to model. It is important to mention that the port used for TLS and SIP is port 5061; therefore, our devices must use TCP/UDP port 5061. After our devices are registered and we can call each other, we can be sure this configuration is working. If we issue the command asterisk -rx 'sip show peer 7003', we will see that the encryption is enabled. At this point, we've just enabled the encryption at the SIP signaling level. With this, we can block any unauthorized user depending on which port the media (voice or/and video) is being transported or steal a username or password or eavesdrop a conversation. Now, we will proceed to enable the audio/video (RTP) encryption. This term is also known as Secure Real Time Protocol (SRTP). To do this, we only enable on the SIP peers the encryption=yes option. The screenshot after this shows an SRTP call between peers 7002 and 7003. This information can be displayed with the command: asterisk -rx 'sip show channel [the SIP channel of our call] The line RTP/SAVP informs us that the call is secure, and the call in the softphone shows an icon with the form of a lock confirming that the call is secure. The following screenshot shows the icon of a lock, informing us that the current call is secured through SRTP: We can have the SRTP enabled without enabling TLS, and we can even activate TLS support between SIP trunks and our Elastix system. There is more… To enable the IAX encryption in our extensions and IAX trunks, add the following line to their configuration file (/etc/asterisk/iax_general_ custom.conf): encryption=aes128 Reload the IAX module with the command: iax2 reload If we would like to see the encryption in action, configure the debug output in the logger.conf file and issue the following CLI commands: CLI> set debug 1 Core debug is at least 1 CLI> iax2 debug IAX2 Debugging Enabled Generating system backups Generating system backups is a very important activity that helps us to restore our system in case of an emergency or failure. The success of our Elastix platform depends on how quickly we can restore our system. In this recipe, we will cover the generation of backups. How to do it… To perform a backup on our Elastix UCS, go to the System | Backup/Restore menu. When entering this module, the first screen that we will see shows all the backup files available and stored in our system, the date they have been created, and the possibility to restore any of them. If we click on any of them, we can download it on to our laptop, tablet, or any device that will allow us to perform a full backup restore, in the event of a disaster. The next screenshot shows the list of backups available on a system. If we select a backup file from the main view, we can delete it by clicking on the Delete button. To create a backup, click on the Perform a Backup button. Select what modules (with their options) will be saved. Click on the Process button to start the backup process on our Elastix box. When done, a message will be displayed informing us that the process has been completed successfully. We can automate this process by clicking on Set Automatic Backup after selecting this option when this process will be started: Daily, Weekly, or Monthly. Restoring a backup from one server to another If we have a backup file, we can copy it to another recently installed Elastix Unified Communications Server, if we'd like to restore it. For example, Server A is a production server, but we'd like to use a brand new server with more resources (Server B). How to do it… After having Elastix installed in Server B, perform a backup, irrespective of whether there is no configuration in it and create a backup in Server A as well. Then, we copy the backup (*.tar file) from Server A to Server B with the console command (being in Server A's console): scp /var/www/backup/back-up-file.tar root@ip-address-of-server-b:/var/www/backup/ Log into Server B's console and change the ownership of the backup file with the command: chown asterisk:asterisk /var/www/backup/back-up-file.tar Restore the copied backup in Server B by using the System | Backup/Restore menu. When this process is being done, Elastix's webGUI will alert us of a restoring process being performed and it will show if there is any software difference between the backup and our current system. We recommend the use of the same Admin and Root passwords and the same telephony hardware in both servers. After this operation is done, we have to make sure that all configurations are working on the new server, before going on production. There is more… If we click on the FTP Backup option, we can drag and drop any selected backup to upload it to a remote FTP server or we can download it locally. We only need to set up the correct data to log us into the remote FTP server. The data to enter are as follows: Server FTP: IP address or domain name of the remote FTP server Port: FTP port User: User Password: Password Path Server FTP: Folder or directory to store the backup The next screenshot shows the FTP-Backup menu and options: Although securing systems is a very important and sometimes difficult area that requires a high level of knowledge, in this article, we discussed the most common but effective tasks that should be done in order to keep your Elastix Unified Communications System healthy and secure. Summary The main objective of this article is to give you all the necessary tools to configure and support an Elastix Unified Communications Server. We will look at these tools through Cookbook recipes, just follow the steps to get an Elastix System up and running. Although a good Linux and Asterisk background is required, this article is structured to help you grow from a beginner to an advanced user. Resources for Article: Further resources on this subject: Lync 2013 Hybrid and Lync Online [article] Creating an Apache JMeter™ test workbench [article] Innovation of Communication and Information Technologies [article]

How does AGI work Let's examine the following diagram: As the previous diagram illustrates, an AGI script communicates with Asterisk via two standard data streams—STDIN (Standard Input) and STDOUT (Standard Output). From the AGI script point-of-view, any input coming in from Asterisk would be considered STDIN, while output to Asterisk would be considered as STDOUT. The idea of using STDIN/STDOUT data streams with applications isn't a new one, even if you're a junior level programmer. Think of it as regarding any input from Asterisk with a read directive and outputting to Asterisk with a print or echo directive. When thinking about it in such a simplistic manner, it is clear that AGI scripts can be written in any scripting or programming language, ranging from BASH scripting, through PERL/PHP scripting, to even writing C/C++ programs to perform the same task. Let's now examine how an AGI script is invoked from within the Asterisk dialplan: exten => _X.,1,AGI(some_script_name.agi,param1,param2,param3) As you can see, the invocation is similar to the invocation of any other Asterisk dialplan application. However, there is one major difference between a regular dialplan application and an AGI script—the resources an AGI script consumes.While an internal application consumes a well-known set of resources from Asterisk, an AGI script simply hands over the control to an external process. Thus, the resources required to execute the external AGI script are now unknown, while at the same time, Asterisk consumes the resources for managing the execution of the AGI script.Ok, so BASH isn't much of a resource hog, but what about Java? This means that the choice of programming language for your AGI scripts is important. Choosing the wrong programming language can often lead to slow systems and in most cases, non-operational systems. While one may argue that the underlying programming language has a direct impact on the performance of your AGI application, it is imperative to learn the impact of each. To be more exact, it's not the language itself, but more the technology of the programming language runtime that is important. The following table tries to distinguish between three programming languages' families and their applicability to AGI development.

The beauty of building a modern web application is being able to take advantage of functionalities such as a Progressive Web App (PWA)! But they can be a little complicated to work with. As always, the Create React App tool makes a lot of this easier for us but does carry some significant caveats that we'll need to think about. This article is taken from the book  Create React App 2 Quick Start Guide by Brandon Richey. This book is intended for those that want to get intimately familiar with the Create React App tool. It covers all the commands in Create React App and all of the new additions in version 2.  To follow along with the examples implemented in this article, you can download the code from the book’s GitHub repository. In this article, we will learn what exactly PWAs are and how we can configure our Create React App project into a custom PWA. We will also explore service workers, their life cycle, and how to use them with Create React App. Understanding and building PWAs Let's talk a little bit about what a PWA is because there is, unfortunately, a lot of misinformation and confusion about precisely what a PWA does! In very simple words, it's simply a website that does the following: Only uses HTTPS Adds a JSON manifest (a web app manifest) file Has a Service Worker A PWA, for us, is a React application that would be installable/runnable on a mobile device or desktop. Essentially, it's just your app, but with capabilities that make it a little more advanced, a little more effective, and a little more resilient to poor/no internet. A PWA accomplishes these via a few tenets, tricks, and requirements that we'd want to follow: The app must be usable by mobile and desktop-users alike The app must operate over HTTPS The app must implement a web app JSON manifest file The app must implement a service worker Now, the first one is a design question. Did you make your design responsive? If so, congratulations, you built the first step toward having a PWA! The next one is also more of an implementation question that's maybe not as relevant to us here: when you deploy your app to production, did you make it HTTPS only? I hope the answer to this is yes, of course, but it's still a good question to ask! The next two, though, are things we can do as part of our Create React App project, and we'll make those the focus of this article. Building a PWA in Create React App Okay, so we identified the two items that we need to build to make this all happen: the JSON manifest file and the service worker! Easy, right? Actually, it's even easier than that. You see, Create React App will populate a JSON manifest file for us as part of our project creation by default. That means we have already completed this step! Let's celebrate, go home, and kick off our shoes, because we're all done now, right? Well, sort of. We should take a look at that default manifest file because it's very unlikely that we want our fancy TodoList project to be called "Create React App Sample". Let's take a look at the manifest file, located in public/manifest.json: { "short_name": "React App", "name": "Create React App Sample", "icons": [ { "src": "favicon.ico", "sizes": "64x64 32x32 24x24 16x16", "type": "image/x-icon" } ], "start_url": ".", "display": "standalone", "theme_color": "#000000", "background_color": "#ffffff" } Some of these keys are pretty self-explanatory or at least have a little bit of information that you can infer from them as to what they accomplish. Some of the other keys, though, might be a little stranger. For example, what does "start_url" mean? What are the different options we can pick for display? What's a "theme_color" or "background_color"? Aren't those just decided by the CSS of our application? Not really. Let's dive deeper into this world of JSON manifest files and turn it into something more useful! Viewing our manifest file in action with Chrome First, to be able to test this, we should have something where we can verify the results of our changes. We'll start off with Chrome, where if you go into the Developer tools section, you can navigate to the Application tab and be brought right to the Service Workers section! Let's take a look at what it all looks like for our application: Exploring the manifest file options Having a manifest file with no explanation of what the different keys and options mean is not very helpful. So, let's learn about each of them, the different configuration options available to us, and some of the possible values we could use for each. name and short_name The first key we have is short_name. This is a shorter version of the name that might be displayed when, for example, the title can only display a smaller bit of text than the full app or site name. The counterpart to this is name, which is the full name of your application.  For example: { "short_name": "Todos", "name": "Best Todoifier" } icons Next is the icons key, which is a list of sub-objects, each of which has three keys. This contains a list of icons that the PWA should use, whether it's for displaying on someone's desktop, someone's phone home screen, or something else. Each "icon" object should contain an "src", which is a link to the image file that will be your icon. Next, you have the "type" key, which should tell the PWA what type of image file you're working with. Finally, we have the "sizes" key, which tells the PWA the size of the icon. For best results, you should have at least a "512x512" and a "192x192" icon. start_url The start_url key is used to tell the application at what point it should start in your application in relation to your server. While we're not using it for anything as we have a single page, no route app, that might be different in a much larger application, so you might just want the start_url key to be something indicating where you want them to start off from. Another option would be to add a query string on to the end of url, such as a tracking link. An example of that would be something like this: { "start_url": "/?source=AB12C" } background_color This is the color used when a splash screen is displayed when the application is first launched. This is similar to when you launch an application from your phone for the first time; that little page that pops up temporarily while the app loads is the splash screen, and background_color would be the background of that. This can either be a color name like you'd use in CSS, or it can be a hex value for a color. display The display key affects the browser's UI when the application is launched. There are ways to make the application full-screen, to hide some of the UI elements, and so on. Here are the possible options, with their explanations: ValueDescription.browserA normal web browser experience.fullscreenNo browser UI, and takes up the entire display.standaloneMakes the web app look like a native application. It will run in its own window and hides a lot of the browser UI to make it look and feel more native. orientation If you want to make your application in the landscape orientation, you would specify it here. Otherwise, you would leave this option missing from your manifest: { "orientation": "landscape" } scope Scope helps to determine where the PWA in your site lies and where it doesn't. This prevents your PWA from trying to load things outside where your PWA runs. start_url must be located inside your scope for it to work properly! This is optional, and in our case, we'll be leaving it out. theme_color This sets the color of the toolbar, again to make it feel and look a little more native. If we specify a meta-theme color, we'd set this to be the same as that specification. Much like background color, this can either be a color name, as you'd use in CSS, or it can be a hex value for a color. Customizing our manifest file Now that we're experts on manifest files, let's customize our manifest file! We're going to change a few things here and there, but we won't make any major changes. Let's take a look at how we've set up the manifest file in public/manifest.json: { "short_name": "Todos", "name": "Best Todoifier", "icons": [ { "src": "favicon.ico", "sizes": "64x64 32x32 24x24 16x16", "type": "image/x-icon" } ], "start_url": "/", "display": "standalone", "theme_color": "#343a40", "background_color": "#a5a5f5" } So we've set our short_name and name keys to match the actual application. We've left the icons key alone completely since we don't really need to do much of anything with that anyway. Next, we've changed start_url to just be "/", since we're working under the assumption that this application is the only thing running on its domain. We've set the display key to standalone since we want our application to have the ability to be added to someone's home screen and be recognized as a true PWA. Finally, we set the theme color to #343a40, which matches the color of the nav bar and will give a more seamless look and feel to the PWA. We also set the background_color key, which is for our splash screen, to #a5a5f5, which is the color of our normal Todo items! If you think back to the explanation of keys, you'll remember we also need to change our meta-theme tag in our public/index.html file, so we'll open that up and quickly make that change: <meta name="theme-color" content="#343a40" /> And that's it! Our manifest file has been customized! If we did it all correctly, we should be able to verify the changes again in our Chrome Developer tools: Hooking up service workers Service workers are defined as a script that your browser runs behind the scenes, separate from the main browser threads. It can intercept network requests, interact with a cache (either storing or retrieving information from a cache), or listen to and deliver push messages. The service worker life cycle The life cycle for a service worker is pretty simple. There are three main stages: Registration Installation Activation Registration is the process of letting the browser know where the service worker is located and how to install it into the background. The code for registration may look something like this: if ('serviceWorker' in navigator) { navigator.serviceWorker.register('/service-worker.js') .then(registration => { console.log('Service Worker registered!'); }) .catch(error => { console.log('Error registering service worker! Error is:', error); }); } Installation is the process that happens after the service worker has been registered, and only happens if the service worker either hasn't already been installed, or the service worker has changed since the last time. In a service-worker.js file, you'd add something like this to be able to listen to this event: self.addEventListener('install', event => { // Do something after install }); Finally, Activation is the step that happens after all of the other steps have completed. The service worker has been registered and then installed, so now it's time for the service worker to start doing its thing: self.addEventListener('activate', event => { // Do something upon activation }); How can we use a service worker in our app? So, how do we use a service worker in our application? Well, it's simple to do with Create React App, but there is a major caveat: you can't configure the service-worker.js file generated by Create React App by default without ejecting your project! Not all is lost, however; you can still take advantage of some of the highlights of PWAs and service workers by using the default Create React App-generated service worker. To enable this, hop over into src/index.js, and, at the final line, change the service worker unregister() call to register() instead: serviceWorker.register(); And now we're opting into our service worker! Next, to actually see the results, you'll need to run the following: $ yarn build We'll create a Production build. You'll see some output that we'll want to follow as part of this: The build folder is ready to be deployed. You may serve it with a static server: yarn global add serve serve -s build As per the instructions, we'll install serve globally, and run the command as instructed: $ serve -s build We will get the following output: Now open up http://localhost:5000 in your local browser and you'll be able to see, again in the Chrome Developer tools, the service worker up and running for your application: Hopefully, we've explored at least enough of PWAs that they have been partially demystified! A lot of the confusion and trouble with building PWAs tends to stem from the fact that there's not always a good starting point for building one. Create React App limits us a little bit in how we can implement service workers, which admittedly limits the functionality and usefulness of our PWA. It doesn't hamstring us, by any means, but doing fun tricks with pre-caching networks and API responses, and loading up our application instantly, even if the browser doing the loading is offline in the first place. That being said, it's like many other things in Create React App: an amazing stepping stone and a great way to get moving with PWAs in the future! If you found this post useful, do check out the book, Create React App 2 Quick Start Guide. In addition to getting familiar with Create React App 2, you will also build modern, React projects with, SASS, and progressive web applications. ReactOS 0.4.11 is now out with kernel improvements, manifests support, and more! React Native community announce March updates, post sharing the roadmap for Q4 React Native Vs Ionic: Which one is the better mobile app development framework?

Getting started with Moodle tests To start with, we need to select a topic or theme for our test. We are going to choose general science, since the subject matter will be easy to incorporate each of the item types we have seen previously. Now that we have an idea of what our topic is going to be, we will get started in the creation of the test. We will be creating all new questions for this test, which will give us the added benefit of a bit more practice in item creation. So, let's get started and work on making our first real test! Let's open our Moodle course, go to the Activity drop-down, and select Create a new Quiz. Once it has been selected, we will be taken to the Quiz creation page and we'll be looking at the General section. The General section Here need to give the test a name that describes what the test is going to cover. Let's call it 'General Science Final Exam' as it describes what we will be doing in the test. The introduction is also important.this is a test students will take and an effective description of what they will be doing is an important point for them. It helps get their minds thinking about the topic at hand, which can help them prepare, and a person who is prepared can usually perform better. For our introduction, we will write the following, 'This test will see how much you learned in our science class this term. The test will cover all the topics we have studied, including, geology, chemistry, biology, and physics. In this test, there are a variety of question types (True/False, Matching, and others). Please look carefully at the sample questions before you move on. If you have any questions during the test, raise your hand. You will have 'x' attempts with the quiz. We have now given the test an effective name and we have given the students a description of what the test will cover. This will be shown in the Info tab to all the students before they take the test, and if we want in the days running up to the test. That's all we need to do in this section. Timing In this section, we need to make some decisions about when we are going to give the test to the students. We will also need to make a decision about how long we will give the students to complete the test. These are important decisions, and we need to make sure we give our students enough time to complete the test. The default Timing section is shown in the next screenshot: We probably know when our final exam will be. So, when we are creating the test, we can set the date that the test will be available to the students and the date it will stop being accessible to them. Because this is our final exam, we only want it to be available for one day, for a specified time period. We will start by clicking on the Disable checkboxes next to Open the Quiz and Close the Quiz dates. This step will enable the date/time drop-down menus and allow us to set them for the test. For us, our test will start on March 20, 2010 at 16:55 p.m. and it will end the same day, one hour later. So we will change the appropriate menus to reflect our needs. If these dates are not set, a student in the course will be able to take the quiz any time after you finish creating it. We will need to give the students time to get in class, settle down, and have their computers ready. However, we also need to make sure the students finish the test in our class, so we have decided to create a time limit of 45 minutes. This means that the test will be open for one hour, and in that one hour time frame, once they start the test, they will have 45 minutes to finish it. To do this, we need to click on the Enable checkbox next to the Time Limit (minutes) textbox. Clicking on this will enable the textbox, and in it we will enter 45. This value will limit the quiz time to 45 minutes, and will show a floating, count-down timer in the test, causing it to auto-submit 45 minutes after it is started. It is good to note that many students get annoyed by the floating timer and its placement on the screen. The other alternative is to have the test proctor have the students submit the quiz at a specified time. Now, we have decided to give a 45 minute time limit on the test, but without any open-ended questions, the test is highly unlikely to take that long. There is also going to be a big difference in the speed at which different students work. The test proctor should explain to the students how much time they should spend on each question and reviewing their answers. Under the Time Limit (minutes) we see the Time delay between first and second attempt and Time delay between later attempts menus. If we are going to offer the test more than once, we can set these, which would force the students to wait until they could try again. The time delays range from 30 minutes to 7 days, and the None setting will not require any waiting between attempts on the quiz. We are going to leave these set to None because this is a final exam and we are only giving it once. Once all the information has been entered into the Timing section, this dialog box is what we have, as shown in the next screenshot: Display Here, we will make some decisions about the way the quiz will look to the students. We will be dividing questions over several pages, which we will use to create divisions in the test. We will also be making decisions about the shuffle questions and shuffle within questions here. Firstly, as the test creators, we should already have a rough idea of how many questions we are going to have on the test. Looking at the Questions Per Page drop-down menu, we have the option of 1 to 50 questions per page. We have decided that we will be displaying six questions per page on the test. Actually, we will only have five questions the students will answer, but we also want to include a description and a sample question for the students to see how the questions look and how to answer them' thus we will have six on each page. We have the option to shuffle questions within pages and within questions. By default, Shuffle Questions is set to No and Shuffle within Questions is set to Yes. We have decided that we want to have our questions shuffled. But wait, we can't because we are using Description questions to give examples, and if we chose shuffle, these examples would not be where they need to be. So, we will leave the Shuffle Questions setting at the default No. However, we do want to shuffle the responses within the question, which will give each student a slightly different test using the same questions and answers. When the display settings are finished, we can see the output shown in the next screenshot: Attempts In this section, we will be setting the number of attempts possible and how further attempts are dealt with. We will also make a decision about the Adaptive Mode. Looking at the Attempts allowed drop-down menu, we have the option to set the number from 1 to 10 or we can set it to Unlimited attempts. For our test, we have already decided to set the value to 1 attempt, so we will select 1 from the drop-down menu. We have the option of setting the Each Attempt Builds on the Last drop-down menu to Yes or No. This feature does nothing now, because we have only set the test to have a single attempt. If we had decided to allow multiple attempts, a Yes setting would have shown the test taker all the previous answers, as if the student were taking the test again, as well as indicating whether he or she were correct or not. If we were giving our students multiple attempts on the test, but we did not want them to see their previous answers, we would set this to No. We are also going to be setting Adaptive mode to No. We do not want our students to be able to immediately see or correct their responses during the test; we want the students to review their answers before submitting anything. However, if we did want the students to check their answers and correct any mistakes during the test, we would set the Attempts Allowed to a number above 1 and the Adaptive Mode to Yes, which would give us the small Submit button where the students would check and correct any mistakes after each question. If multiple attempts are not allowed, the Submit button will be just that, a button to submit your answer. Here is what the Attempts section looks like after we have set our choices: Grades In this section, we will set the way Moodle will score the student. We see three choices in this section, Grading method, Apply penalties, and Decimal digits in grades; however, because we have only selected a single attempt, two of these options will not be used. Grading Method allows us to determine which of the scores we want to give our student after multiple tries. We have four options here: Highest Grade, Average Grade, First Attempt, and Last Attempt. Highest Grade uses the highest grade achieved from any attempt on any individual question. The Average Grade will take the total number of tries and grades and average them. The First Attempt will use the grade from the first attempt and the Last Attempt will use the grade from the final attempt. Since we are only giving one try on our test, this setting has no function and we will leave it set at its default, Highest Grade, because either option would give the same result. Apply penalties is similar to Grading method, in that it does not function because we have turned off Adaptive Mode. If we had set Adaptive Mode to Yes, then this feature would give us the option of applying penalties, which are set in the individual question setup pages. If we were using Adaptive Mode and this option feature set to No, then there would be no penalties for mistakes as in previous attempts. If it were set to Yes, the penalty amount decided on in the question would be subtracted for each incorrect response from the total points available on the question. However, our test is not set to Adaptive Mode, so we will leave it at the default setting, Yes. It is important to note here that no matter how often a student is penalized for an incorrect response, their grade will never go below zero. The Decimal digits in grades shows the final grade the student receives with the number of decimal places selected here. There are four choices available in this setting: 0, 1, 2, and 3. If, for example, the number is set to 1, the student will receive a score calculated to 1 decimal place, and the same follows for 2 and 3. If the number is set to 0, the final score will be rounded. We will set our Decimal digits in grades to 0. After we have finished, the Grades section appears as shown in the next screenshot: Review options This sectopm is where we set when and what our students will see when they look back at the test. There are three categories: Immediately after the attempt; Later, while quiz is still open; and After the quiz is closed. The first category, Immediately after the attempt, will allow students to see whatever feedback we have selected to display immediately after they click on the Submit all and finish button at the end of the test, or Submit, in the case of Adaptive mode. The second category, Later, while quiz is still open, allows students to view the selected review options any time after the test is finished, that is, when no more attempts are left, but before the test closes. Using the After the quiz is closed setting will allow the student to see the review options after the test closes, meaning that students are no longer able to access the test because a close date was set. The After the quiz is closed option is only useful if a time has been set for the test to close, otherwise the review never happens because the test doesn't ever close. Each of these three categories contains the same review options: Responses, Answers, Feedback, General feedback, Scores, and Overall feedback.Here is what these options do: Responses are the student's response to the question and whether he or she were wrong or correct. Answers are the correct response to the question. Feedback is the feedback you enter based on the answer the student gives. This feedback is different from the General quiz feedback they may receive. General feedback are the comments all students receive, regardless of their answers. Scores are the scores the student received on the questions. Overall feedback are the comments based on the overall grade on the test. We want to give our students all of this information, so they can look it over and find out where they made their mistakes, but we don't want someone who finishes early to have access to all the correct answers. So, we are going to eliminate all feedback on the test until after it closes. That way there is no possibility for the students to see the answers while other students might still be taking the test. To do remove such feedback, we simply unclick all the options available in the categories we don't want. Here is what we have when we are finished: Regardless of the options and categories we select in the Review options, students will always be able to see their overall scores. Looking at our settings, the only thing a student will be able to view immediately after the test is complete is the score. Only after the test closes, will the student be able to see the full range of review material we will be providing. If we had allowed multiple attempts, we would want to have different settings. So, instead of After the quiz is closed, we would want to set our Review options to Immediately after the attempt, because this setting would let the student know where he or she had problems and which areas of the quiz need to be focussed on. One final point here is that even a single checkbox in any of the categories will allow the student to open and view the test, giving the selected review information to the student. This option may or may not be what you want. Be careful to ensure that you have only selected the options and categories you want to use. Security This section is where we can increase quiz security, but it is important to note that these settings will not eliminate the ability of tech-savvy students to cheat. What this section does is provide a few options that make cheating a bit more difficult to do. We have three options in this section: Browser security, Require password, and Require network address. The Browser security drop-down has two options: None and Full screen popup with some JavaScript security. The None option is the default setting and is appropriate for most quizzes. This setting doesn't make any changes in browser security and is the setting you will most likely want to use for in-class quizzes, review quizzes, and others. Using the fullscreen option will create a browser that limits the options for students to fiddle things. This option will open a fullscreen browser window with limited navigation options. In addition to limiting the number of navigation options available, this option will also limit the keyboard and mouse commands available. This option is more appropriate for high-stakes type tests and shouldn't be used unless there is a reason. This setting also requires that JavaScript is used. Browser security is more a safety measure against students pressing the wrong button than preventing cheating, but can help reduce it. The Require password does exactly what you think it would. It requires the students to enter a password before taking the test. To keep all your material secure, I recommend using a password for all quizzes that you create. This setting is especially important if you are offering different versions of the quiz to different classes or different tests in the same class and you want to make sure only those who should be accessing the quiz can. There is also an Unmask checkbox next to the password textbox. This option will show you the password, just in case you forget! Finally, we have the Require network address option, which will only allow those at certain IP Addresses to access the test. These settings can be useful to ensure that only students in the lab or classroom are taking the test. This setting allows you to enter either complete IP Addresses (for example. 123.456.78.9), which require that specific address to begin the test; partial IP Addresses (for example 123.456), which will accept any address as long as it begins with the address prefixes; and what is known as Classless Inter-Domain Routing (CIDR) notation, (for example 123.456.78.9/10), which only allows specific subnets. You might want to consult with your network administrator if you want to use this security option. By combining these settings, we can attempt to cut down on cheating and improper access to our test. In our case here, we are only going to use the fullscreen option. We will be giving the test in our classroom, using our computers, so there is no need to turn on the IP Address function or require a password. When we have finished, the Security section appears as shown in the next screenshot:

  Alfresco 3 Records Management Comply with regulations and secure your organization’s records with Alfresco Records Management. Successfully implement your records program using Alfresco Records Management, fully certified for DoD-5015.2 compliance The first and only book to focus exclusively on Alfresco Records Management Step-by-step instructions describe how to identify records, organize records, and manage records to comply with regulatory requirements Learn in detail about the software internals to get a jump-start on performing customizations Appendix         Read more about this book       (For more resources on this subject, see here.) The steering committee To succeed, our Records Management program needs continued commitment from all levels of the organization. A good way to cultivate that commitment is by establishing a steering committee for the records program. From a high level, the steering committee will direct the program, set priorities for it, and assist in making decisions. The steering committee will provide the leadership to ensure that the program is adequately funded, staffed, properly prioritized with business objectives, and successfully implemented. Committee members should know the organization well and be in a position to be both able and willing to make decisions. Once the program is implemented, the steering committee should not be dissolved; it still will play an important function. It will continue to meet and oversee the Records Management program to make sure that it is properly maintained and updated. The Records Management system is not something that can simply be turned on and forgotten. The steering committee should meet regularly, track the progress of the implementation, keep abreast of changes in regulatory controls, and be proactive in addressing the needs of the Records Management program. Key stakeholders The Records Management steering committee should include executives and senior management from core business units such as Compliance, Legal, Finance, IT, Risk Management, Human Resources, and any other groups that will be affected by Records Management. Each of these groups will represent the needs and responsibilities of their respective groups. They will provide input relative to policies and procedures. The groups will work together to develop a priority-sequenced implementation plan that all can agree upon. Creating a committee that is heavily weighted with company executives will visibly demonstrate that our company is strongly committed to the program and it ensures that we will have the right people on board when it is time to make decisions, and that will keep the program on track. The steering committee should also include representatives from Records Management, IT, and users. Alternatively, representatives from these groups can be appointed and, if not members of the steering committee, they should report directly to the steering committee on a regular basis: The Program Contact The Program Contact is the chair of the steering committee. This role is typically held by someone in senior management and is often someone from the technology side of the business, such as the Director of IT. The Program Contact signs off with the final approval on technology deliverables and budget items. The Program Sponsor A key member of the records steering committee is the Program Sponsor or Project Champion. This role is typically held by a senior executive who will be able to represent the records initiative within the organization's executive team. The Sponsor will be able to establish the priority of the records program, relative to other organizational initiatives and be able to persuade the executive team and others in the company of the importance of the records management initiative. Corporate Records Manager Another key role of the steering committee is the Corporate Records Manager. This role acts as the senior champion for the records program and is responsible for defining the procedures and policies around Records Management. The person in this role will promote the rollout of and the use of the records program. They will work with each of the participating departments or groups, cultivating local champions for Records Management within each of those groups. The Corporate Records Manager must effectively communicate with business units to explain the program to all staff members and work with the various business units to collect user feedback so that those ideas can be incorporated into the planning process. The Corporate Records Manager will try to minimize any adverse user impact or disruption. Project Manager The Project Manager typically is on the steering committee or reports directly to it. The Project Manager plans and tracks the implementation of work on the program and ensures that program milestones are met. The person in this role manages both, the details of the system setup and implementation. This Project Manager also manages the staff time spent working on the program tasks. Business Analyst The Business Analyst analyzes business processes and records, and from these, creates a design and plan for the records program implementation. The Business Analyst works closely with the Corporate Records Manager to develop records procedures and provides support for the system during rollout. Systems Administrator The Systems Administrator leads the technical team for supporting the records application. The Systems Administrator specifies and puts into place the hardware required for the records program, the storage space, memory, and CPU capabilities. The person in this role monitors the system performance and backs up the system regularly. The Systems Administrator leads the team to apply software upgrades and to perform system troubleshooting. The Network Administrator The Network Administrator ensures that the network infrastructure is in place for the records program to support the appropriate bandwidth for the server and client workstations that will access the application. The Network Administrator works closely with the Systems Administrator. The Technical Analyst The Technical Analyst is responsible for analyzing the configuration of the records program. The Technical Analyst needs to work closely with the Business Analyst and Corporate Records Manager. The person in this role will specify the classification and structure used for the records program File Plan. They will also specify the classes of documents stored as records in the records application and the associated metadata for those documents. The Records Assistant The Records Assistant assists in the configuration of the records application. Tasks that the Records Assistant will perform include data entry and creating the folder structure hierarchy of the File Plan within the records application based on the specification created by the Technical Analyst. The Records Developer The Records Developer is a software engineer that is assigned to support the implementation of the records program, based on requirements derived by the Business Analyst. The Records Developer may need to edit and update configuration files, often using technologies like XML. The Records Developer may also need to make customizations to the user interface of the application. The Trainer The Trainer will work with end users to ensure that they understand the system and their responsibilities in interacting with it. The trainer typically creates training materials and provides training seminars to users. The Technical Support Specialist The Technical Support Specialist provides support to users on the functioning of the Records Management application. This person is typically an advanced user and is trained to be able to provide guidance in interacting with the application. But more than just the Records Management application, the support specialist should also be well versed in and be able to assist users and answer their questions about records processes and procedures, as well as concepts like retention and disposition of documents. The Technical Support Specialist will, very often, be faced with requests or questions that are really enhancement requests. The support specialist needs to have a good understanding of the scope of the records implementation and be able to distinguish an enhancement request from a defect or bug report. Enhancements should be collected and routed back through the Project Manager and, depending on the nature of the request or problem, possibly even to the Corporate Records Manager or the Steering Committee. Similarly, application defects or bugs that are found should be reported back through to the Project Manager. Bug reports will be prioritized by the Project Manager, as appropriate, assigned to the Technical Developers, or reported to the Systems Integrator or to Alfresco. The Users The Users are the staff members who will use the Records Management application as part of their job. Users are often the key to the success or failure of a records program. Unfortunately, users are one aspect of the project that is often overlooked. Obviously, it is important that the records application be well designed and meet the objectives and requirements set out for it. But if users complain and can't accept it, then the program will be doomed to failure. Users will often be asked to make changes to processes that they have become very comfortable with. Frequent and early communication with users is a must in order to ultimately gain their acceptance and participation. Prior to and during the implementation of the records system, users should receive status updates and explanations from the Corporate Records Manager and also from the Records Manager lead in their business unit. It is important that frequent communications be made with users to ensure their opinions and ideas are heard, and also so that they will learn to be able to most effectively use the records system. Once the application is ready, or better yet, well before the application goes live, users should attend training sessions on proper records-handling behavior; they should experience hands-on training with the application; and they should also be instructed in how best to communicate with the Technical Support Specialist, should they ever have questions or encounter any problems. Alfresco, Consultants, and Systems Integrators Alfresco is the software vendor for Alfresco Records Management, but Alfresco typically does not work directly with customers. We could go at it alone, but more likely, we'll probably choose to work directly with one of Alfresco's System Integration partners or consultants in planning for and setting up our system. Depending on the size of our organization and the available skill set within it, the Systems Integrator can take on as much or as little of the burden for helping us to get up and running with our Records Management program. Almost any of the Technical Team roles discussed in this section, like those of the Analyst and Developer, and even the role of the Project Manager, are ones that can be performed by a Systems Integrator. A list of certified Alfresco Integrators can be found on the Alfresco site: http://www.alfresco.com/partners/search.jsp?t=si A Systems Integrator can bring to our project an important breadth of experience that can help save time and ensure that our project will go smoothly. Alfresco Systems Integration partners know their stuff. They are required to be certified in Alfresco technology and they have worked with Alfresco extensively. They are familiar with best practices and have picked up numerous implementation tips and tricks having worked on similar projects with other clients.

PowerShell is Microsoft’s automation platform for IT Pros. Of late, there have been a lot of questions around the complexity of this latest automation tool by Microsoft. At Microsoft Ignite 2018, Jason Himmelstein, Director of Technical Strategy and Strategic Partnerships, Office Apps & Services MVP, explained the basics of PowerShell and how to truly optimize your SharePoint implementation using this powerful IT pro toolset. While in this post we look at the big picture, you can check out the complete video here: ‘Introduction to PowerShell for the anxious IT pro’. Want to do more with PowerShell? After learning the basics, you can learn how to use PowerShell to automate complex Windows server tasks. You can also improve PowerShell's usability, and control and manage Windows-based environments by working through exciting recipes given in Windows Server 2019 Automation with PowerShell Cookbook - Third Edition written by Thomas Lee.  Himmelstein starts off by saying PowerShell isn’t a packaged executable, nor it is developer-centric that needs one to understand code, and it is easy for an IT Pro to understand. What is PowerShell? Windows PowerShell is Microsoft’s task automation framework, consisting of a command-line shell and associated scripting language built on .NET Framework. It provides full access to COM and WMI, enabling administrators to perform administrative tasks on both local and remote Windows systems. In simple words, PowerShell is an object-based, not a text-based, command-line interface for Microsoft Technologies. This means results in PowerShell can be acted upon and not just read from. One can cause huge damage to an environment using PowerShell as there is no back button in PowerShell. However, to check what must have gone wrong, you can check the logs but can not undo actions. Why PowerShell matters Regardless of the platform a person uses such as Office 365, Azure, etc., PowerShell can be easily implemented due to its cross-platform capability. Himmelstein also highlights one can also get started with Azure PowerShell by trying it out in an Azure Cloud Shell environment, an interactive, authenticated, browser-accessible shell for managing Azure resources.  Azure Cloud Shell comes equipped with commonly used CLI tools including Linux shell interpreters, PowerShell modules, Azure tools, text editors, source control, build tools, container tools, database tools and more. Cloud Shell also includes language support for several popular programming languages such as Node.js, .NET and Python. Cloud Shell also securely authenticates automatically for instant access to your resources through the Azure CLI or Azure PowerShell cmdlets. Users can use PowerShell in Cloud Shell. One can also develop applications using PowerShell or can use PowerShell via Source Control Management (SCM). Basics of PowerShell PowerShell Hardware There are two ways one can use PowerShell; one is via the PowerShell Console, which is similar to a command line. The other is PowerShell ISE (Integrated Scripting Environment). One thing Himmelstein encourages is, “we run PowerShell in the Console and we write PowerShell in the ISE.” The reason is there are certain functionalities that do not work in the ISE when one hits the ‘Run’ command. In such cases, the user will have to take that PowerShell out, copy it, save the file and run it in a command window. cmdlets Cmdlets are the main building blocks of PowerShell. These are mini commands that perform one action. These have the ability to pipe the output of one cmdlet into further cmdlets. These can also perform equality tests with expressions such as -eq, -lt, -match; one can diff easily within a PowerShell. Modules There are four types of Modules in PowerShell: Script: A Script module is a file (.psm1) that contains any valid Windows PowerShell code. Binary: A binary module is a .NET framework assembly (.dll) that contains compiled code. Manifest: A module Manifest is a Windows PowerShell data file (.psd1) that describes the contents of a module and determines how a module is processed. Dynamic: A dynamic module does not persist to disk. It is created using New Module, is intended to be short-lived, and cannot be accessed by Get-Module. Himmelstein prefers not to use the Dynamic module as it persists for just one session. Objects and Members Objects are instances of classes and have properties and methods. Members are properties and methods of an object. Properties define what an Object is and Methods define what you can do with the object. Himmelstein puts together all these terms in a simple way: Objects = stuff Cmdlets = things you can do with the stuff Modules = list of things you can do with the stuff Properties = details about the stuff Methods = instructions for things you can do with the stuff PipeLine Using PipeLines one can chain objects together for processing. The output of a pipelined object becomes the object itself. Functional Explanation Get-command: Gets all the cmdlet installed on your computer. Get-help: Displays additional information about a cmdlet Get-member: Listing the Properties and Methods of a Command or Object Get-verb: Gets approved Windows PowerShell verbs Start-transcript: Logs everything you do in that PowerShell window to a file Get- history: If you didn’t start transcript, you can still review your history before closing your Shell or ISE window. Tips for PowerShell beginners Use Variables: You can use any variables except the ones that are reserved by the system, which you will be prompted when you try to enter a reserved variable. Call one thing at a time Comment your scripts as this may save you a lot of time. Create scripts using an ISE/IDE, you can also use the Visual Studio Code and then execute in Shell. Dispose of your objects. Close the command window by typing Exit. Test before using in Production Write reusable scripts. What Powershell beginners should avoid Rewriting your variables Hard coding your scripts such as Password as it may get fired by PowerShell Taking code from the internet or vendor and just Run in your environment (You should read every code before you run it in your environment). Assuming the code is not harmful; it is. There is no back button in PowerShell and you cannot undo things. Running your code in an IDE/ISE and expect everything to work. PowerShell Syntax and Bracketology Syntax ‘#’ is for Comment ‘+’ is for Add ‘=’, ‘-eq’, are for Equal ‘!’, ‘-ne’, ‘-not’ are for ‘not equal’ Brackets ‘()’ Curved brackets also known as Parentheses are used for required options, compulsory arguments, or control structures. ‘{}’ Curly brackets are used for block expression within a command block and is also used to open a code block ‘[]’ Square brackets are used to denote optional elements or parameters and also used for match functions. Now that you know the basics of PowerShell, you can start performing key admin tasks on Windows Server 2019. To further learn how to employ best practices for writing PowerShell scripts and configuring Windows Server 2019 and leverage PowerShell to automate complex Windows server tasks, check out our book, Windows Server 2019 Automation with PowerShell Cookbook - Third Edition written by Thomas Lee. Weaponizing PowerShell with Metasploit and how to defend against PowerShell attacks [Tutorial] Scripting with Windows Powershell Desired State Configuration [Video] Automate tasks using Azure PowerShell and Azure CLI [Tutorial]