How-To Tutorials

(For more resources related to this topic, see here.) Social engineering is an act of manipulating people to perform actions that they don't intend to do. A cyber-based, socially engineered scenario is designed to trap a user into performing activities that can lead to the theft of confidential information or some malicious activity. The reason for the rapid growth of social engineering amongst hackers is that it is difficult to break the security of a platform, but it is far easier to trick the user of that platform into performing unintentional malicious activity. For example, it is difficult to break the security of Gmail in order to steal someone's password, but it is easy to create a socially engineered scenario where the victim can be tricked to reveal his/her login information by sending a fake login/phishing page. The Social-Engineer Toolkit is designed to perform such tricking activities. Just like we have exploits and vulnerabilities for existing software and operating systems, SET is a generic exploit of humans in order to break their own conscious security. It is an official toolkit available at https://www.trustedsec.com/, and it comes as a default installation with BackTrack 5. In this article, we will analyze the aspect of this tool and how it adds more power to the Metasploit framework. We will mainly focus on creating attack vectors and managing the configuration file, which is considered the heart of SET. So, let's dive deeper into the world of social engineering. Getting started with the Social-Engineer Toolkit (SET) Let's start our introductory recipe about SET, where we will be discussing SET on different platforms. Getting ready SET can be downloaded for different platforms from its official website: https://www.trustedsec.com/. It has both the GUI version, which runs through the browser, and the command-line version, which can be executed from the terminal. It comes pre-installed in BackTrack, which will be our platform for discussion in this article. How to do it... To launch SET on BackTrack, start the terminal window and pass the following path: root@bt:~# cd /pentest/exploits/set root@bt:/pentest/exploits/set# ./set Copyright 2012, The Social-Engineer Toolkit (SET) All rights reserved. Select from the menu: If you are using SET for the first time, you can update the toolkit to get the latest modules and fix known bugs. To start the updating process, we will pass the svn update command. Once the toolkit is updated, it is ready for use. The GUI version of SET can be accessed by navigating to Applications | BackTrack | Exploitation tools | Social-Engineer Toolkit. How it works... SET is a Python-based automation tool that creates a menu-driven application for us. Faster execution and the versatility of Python make it the preferred language for developing modular tools, such as SET. It also makes it easy to integrate the toolkit with web servers. Any open source HTTP server can be used to access the browser version of SET. Apache is typically considered the preferable server while working with SET. There's more... Sometimes, you may have an issue upgrading to the new release of SET in BackTrack 5 R3. Try out the following steps: You should remove the old SET using the following command: dpkg –r set We can remove SET in two ways. First, we can trace the path to /pentest/exploits/set, making sure we are in the directory and then opt for the 'rm' command for removing all files present there. Or, we can use the method shown previously. Then, for reinstallation, we can download its clone using the following command: Git clone https://github.com/trustedsec/social-engineer-toolkit /set Working with the SET config file In this recipe, we will take a close look at the SET config file, which contains default values for different parameters that are used by the toolkit. The default configuration works fine with most of the attacks, but there can be situations when you have to modify the settings according to the scenario and requirements. So, let's see what configuration settings are available in the config file. Getting ready To launch the config file, move to the config file and open the set_config file: root@bt:/pentest/exploits/set# nano config/set_config The configuration file will be launched with some introductory statements, as shown in the following screenshot: How to do it... Let's go through it step-by-step: First, we will see what configuration settings are available for us: # DEFINE THE PATH TO METASPLOIT HERE, FOR EXAMPLE /pentest/exploits/framework3 METASPLOIT_PATH=/pentest/exploits/framework3 The first configuration setting is related to the Metasploit installation directory. Metasploit is required by SET for proper functioning, as it picks up payloads and exploits from the framework: # SPECIFY WHAT INTERFACE YOU WANT ETTERCAP TO LISTEN ON, IF NOTHING WILL DEFAULT # EXAMPLE: ETTERCAP_INTERFACE=wlan0 ETTERCAP_INTERFACE=eth0 # # ETTERCAP HOME DIRECTORY (NEEDED FOR DNS_SPOOF) ETTERCAP_PATH=/usr/share/ettercap Ettercap is a multipurpose sniffer for switched LAN. Ettercap section can be used to perform LAN attacks like DNS poisoning, spoofing etc. The above SET setting can be used to either set ettercap ON of OFF depending upon the usability. # SENDMAIL ON OR OFF FOR SPOOFING EMAIL ADDRESSES SENDMAIL=OFF The sendmail e-mail server is primarily used for e-mail spoofing. This attack will work only if the target's e-mail server does not implement reverse lookup. By default, its value is set to OFF. The following setting shows one of the most widely used attack vectors of SET. This configuration will allow you to sign a malicious Java applet with your name or with any fake name, and then it can be used to perform a browser-based Java applet infection attack: # CREATE SELF-SIGNED JAVA APPLETS AND SPOOF PUBLISHER NOTE THIS REQUIRES YOU TO # INSTALL ---> JAVA 6 JDK, BT4 OR UBUNTU USERS: apt-get install openjdk-6-jdk # IF THIS IS NOT INSTALLED IT WILL NOT WORK. CAN ALSO DO apt-get install sun-java6-jdk SELF_SIGNED_APPLET=OFF We will discuss this attack vector in detail in a later recipe, that is, the Spear phishing attack vector . This attack vector will also require JDK to be installed on your system. Let's set its value to ON, as we will be discussing this attack in detail: SELF_SIGNED_APPLET=ON # AUTODETECTION OF IP ADDRESS INTERFACE UTILIZING GOOGLE, SET THIS ON IF YOU WANT # SET TO AUTODETECT YOUR INTERFACE AUTO_DETECT=ON The AUTO_DETECT flag is used by SET to auto-discover the network settings. It will enable SET to detect your IP address if you are using NAT/Port forwarding, and it allows you to connect to the external Internet. The following setting is used to set up the Apache web server to perform web-based attack vectors. It is always preferred to set it to ON for better attack performance: # USE APACHE INSTEAD OF STANDARD PYTHON WEB SERVERS, THIS WILL INCREASE SPEED OF # THE ATTACK VECTOR APACHE_SERVER=OFF # # PATH TO THE APACHE WEBROOT APACHE_DIRECTORY=/var/www The following setting is used to set up the SSL certificate while performing web attacks. Several bugs and issues have been reported for the WEBATTACK_SSL setting of SET. So, it is recommended to keep this flag OFF: # TURN ON SSL CERTIFICATES FOR SET SECURE COMMUNICATIONS THROUGH WEB_ATTACK VECTOR WEBATTACK_SSL=OFF The following setting can be used to build a self-signed certificate for web attacks, but there will be a warning message saying Untrusted certificate. Hence, it is recommended to use this option wisely to avoid alerting the target user: # PATH TO THE PEM FILE TO UTILIZE CERTIFICATES WITH THE WEB ATTACK VECTOR (REQUIRED) # YOU CAN CREATE YOUR OWN UTILIZING SET, JUST TURN ON SELF_SIGNED_CERT # IF YOUR USING THIS FLAG, ENSURE OPENSSL IS INSTALLED! # SELF_SIGNED_CERT=OFF The following setting is used to enable or disable the Metasploit listener once the attack is executed: # DISABLES AUTOMATIC LISTENER - TURN THIS OFF IF YOU DON'T WANT A METASPLOIT LISTENER IN THE BACKGROUND. AUTOMATIC_LISTENER=ON The following configuration will allow you to use SET as a standalone toolkit without using Metasploit functionalities, but it is always recommended to use Metasploit along with SET in order to increase the penetration testing performance: # THIS WILL DISABLE THE FUNCTIONALITY IF METASPLOIT IS NOT INSTALLED AND YOU JUST WANT TO USE SETOOLKIT OR RATTE FOR PAYLOADS # OR THE OTHER ATTACK VECTORS. METASPLOIT_MODE=ON These are a few important configuration settings available for SET. Proper knowledge of the config file is essential to gain full control over the SET. How it works... The SET config file is the heart of the toolkit, as it contains the default values that SET will pick while performing various attack vectors. A misconfigured SET file can lead to errors during the operation, so it is essential to understand the details defined in the config file in order to get the best results. The How to do it... section clearly reflects the ease with which we can understand and manage the config file. Working with the spear-phishing attack vector A spear-phishing attack vector is an e-mail attack scenario that is used to send malicious mails to target/specific user(s). In order to spoof your own e-mail address, you will require a sendmail server. Change the config setting to SENDMAIL=ON. If you do not have sendmail installed on your machine, then it can be downloaded by entering the following command: root@bt:~# apt-get install sendmail Reading package lists... Done Getting ready Before we move ahead with a phishing attack, it is imperative for us to know how the e-mail system works. Recipient e-mail servers, in order to mitigate these types of attacks, deploy gray-listing, SPF records validation, RBL verification, and content verification. These verification processes ensure that a particular e-mail arrived from the same e-mail server as its domain. For example, if a spoofed e-mail address, <[email protected]>, arrives from the IP 202.145.34.23, it will be marked as malicious, as this IP address does not belong to Gmail. Hence, in order to bypass these security measures, the attacker should ensure that the server IP is not present in the RBL/SURL list. As the spear-phishing attack relies heavily on user perception, the attacker should conduct a recon of the content that is being sent and should ensure that the content looks as legitimate as possible. Spear-phishing attacks are of two types—web-based content and payload-based content. How to do it... The spear-phishing module has three different attack vectors at our disposal: Let's analyze each of them. Passing the first option will start our mass-mailing attack. The attack vector starts with selecting a payload. You can select any vulnerability from the list of available Metasploit exploit modules. Then, we will be prompted to select a handler that can connect back to the attacker. The options will include setting the vnc server or executing the payload and starting the command line, and so on. The next few steps will be starting the sendmail server, setting a template for a malicious file format, and selecting a single or mass-mail attack: Finally, you will be prompted to either choose a known mail service, such as Gmail or Yahoo, or use your own server: 1. Use a gmail Account for your email attack. 2. Use your own server or open relay set:phishing>1 set:phishing> From address (ex: [email protected]):[email protected] set:phishing> Flag this message/s as high priority? [yes|no]:y Setting up your own server cannot be very reliable, as most of the mail services follow a reverse lookup to make sure that the e-mail has generated from the same domain name as the address name. Let's analyze another attack vector of spear-fishing. Creating a file format payload is another attack vector in which we can generate a file format with a known vulnerability and send it via e-mail to attack our target. It is preferred to use MS Word-based vulnerabilities, as they are difficult to detect whether they are malicious or not, so they can be sent as an attachment via an e-mail: set:phishing> Setup a listener [yes|no]:y [-] *** [-] * WARNING: Database support has been disabled [-] *** At last, we will be prompted on whether we want to set up a listener or not. The Metasploit listener will begin and we will wait for the user to open the malicious file and connect back to the attacking system. The success of e-mail attacks depends on the e-mail client that we are targeting. So, a proper analysis of this attack vector is essential. How it works... As discussed earlier, the spear-phishing attack vector is a social engineering attack vector that targets specific users. An e-mail is sent from the attacking machine to the target user(s). The e-mail will contain a malicious attachment, which will exploit a known vulnerability on the target machine and provide a shell connectivity to the attacker. The SET automates the entire process. The major role that social engineering plays here is setting up a scenario that looks completely legitimate to the target, fooling the target into downloading the malicious file and executing it.

Unity assets are part of the unity ecosystem which help you to create in-game environments, gameplay options effectively. In this article, we are going to show you how to work with Unity assets which will eventually help you create fun and interactive 2D games with Unity 2017. This article is a part of the book titled "Unity 2017 2D Game Development Projects" written by Lauren S. Ferro & Francesco Sapio. This book helps you to create exciting 2D games from scratch easily. Textures and Sprites Before you start anything within Unity, it is useful to know that Textures and Sprites within Unity are two separate things, although they are used in similar contexts. To begin, a Sprite is an image that can be used as a 2D object. It has only two coordinates: x-axis and y-axis. Therefore, all the graphical components of 2D game development are called Sprites. Sprites can be repositioned, scaled, and rotated like any other game object in Unity. You can move, destroy, or create it during the game. Sprites, by default, are rendered directly against the camera; however, you can easily change this if you are using the Sprite Renderer in a 3D scene. They work with the Sprite Renderer, unlike a 3D object, which works with the Mesh Renderer. Besides Sprites, there are other graphical components called Textures. These are also images, but they are used to change the appearance of an object in both 2D (for example, Sprites and background) and 3D (for example, an object or character's appearance). But Textures are not objects. This means that you cannot get them to move during gameplay. Saying that, you can create images with Textures that animate, with Sprite Sheets/Atlases. What this means is that each frame of an animation is placed on a Sprite Sheet, which is a Texture, that will eventually be cut up so that each frame of the animation is played sequentially. Throughout we will use the terms Sprite Sheets and Atlases. While they are pretty much the same thing, the subtle difference between the two is that a Sprite Sheet generally has Sprite (frame-by-frame) animations, whereas an Atlas will contain images such as tileable Textures for the walls and other environmental components (for example, objects). Their purpose is to maximize the space by combining multiple images into one Texture, whether for characters (and their animations) or environmental Textures. More generally speaking, when it comes to handling Sprites and Textures, Unity has various tools that deal with them in different ways and are used for different purposes. A brief overview of each of them follows. We will discuss them in more detail: Sprite Editor: This is used to edit Sprites. This is done by selecting them individually from a larger image, known as a Sprite Atlas, or by changing their Pivot point, and so on. Sprite Creator: This is used to create a Sprite placeholder. This is useful if you do not have any Sprites to use but want to continue implementing the functionality of a game. Sprite placeholders can be replaced later with actual Sprites. Sprite Packer: This is used to increase the efficiency of your project's usage of main memory. It achieves this by packing various Sprites into a single place using the Packing Tag attribute. This appears in the Inspector window when you select a Sprite in the Project window. Sprite Render The Sprite Render displays images that have been imported as the type Sprite. There are a number of different parameters within the Sprite Render that allows you to modify a Sprite. We will discuss them here: Color: Color allows you to change the color value and the value of the Alpha channel (transparency) of a Sprite Flip: Flip is what defines the axis that the Sprite needs to be flipped on Material: Material refers to the material that Unity will use to render the Sprite Sorting Layer: Sorting Layer defines which layer the Sprite should be rendered on (it basically indicates the order in which the different Sprites are drawn, for example, which one is on top of the others) Order in Layer: Order in Layer is the order within the Sorting Layer Sprite Editor In some cases, you may have a Texture that contains just one graphic element; in other cases, you may have multiple ones. The latter is more effective for many reasons, such as saving computational resources and keeping things organized. A case in which you are likely to combine many Sprites into one Texture may be frame-by-frame animations of a character, where other Sprites may be parts of a character (such as clothing and items), and will need to be customizable, such as different items (and their effects). In Unity, you can easily extract elements from a single Texture by using the Sprite Editor. The Sprite Editor is used to take multiple elements from an Atlas or Sprite Sheet and slice them into individual Sprites. How to use the Sprite Editor To open the Sprite Editor, perform the following steps: Drag and drop some images (anything you have on your computer, so you can have them as test images) into the Project panel. Select the 2D image you want to edit from the Project view. In the Inspector, change the Texture Type into Sprite (2D and UI), so you will be able to use it within the Sprite Editor. Click on the Sprite Editor button in the Texture Import Inspector and the Sprite Editor displays. When you open the Sprite Editor window, you can move it around like any other window within Unity; you can also dock next to others such as the Hierarchy or Project windows. To select the Sprites, simply click and drag on the Sprite that you wish to select. As a result, you will have bounding boxes around each Sprite element that you have selected, as in the following screenshot: If you happen to click and drag too much around a Sprite, don't panic! You can easily resize the bounding box by clicking on any of the four corners or edges of the bounding box, like in the upcoming screenshot. Alternatively, you can also reposition the bounding box by clicking and dragging in the middle of the box itself. While you're creating these selections, it is important to make sure that you name them appropriately. To do this, click on the box surrounding the Sprite that you wish to name. You will notice that a box appears. Now, next to where it says Name is where you enter the name that you wish to call your Sprite. Another thing that is also to keep in mind here is the Pivot of the Sprite. Think of this as the Sprite's center. For example, if you rotate a Sprite, it will rotate wherever its Pivot is .0. A few more elements that you will also find useful while you are slicing up your Sprites are the options located at the top of the Sprite Editor window. We will discuss them now. You can only see the Sprite Editor button if the TextureType on the image you have selected is set to Sprite (2D and UI). In addition, you cannot edit a Sprite which is in the Scene view. Slice Menu: One great feature of Unity is the opportunity to automatically slice Sprites. What this means is that if you have a large Sprite sheet with various animations, images, and so on, you can automatically cut each image out. You have two options to do this: Automatic: Automatic is better for when you have unevenly distributed Sprites, such as the case with an Atlas.  When choosing the location of the Pivot, it will, by default set it to the center. Method: Method tells you how to deal with existing Sprites within the Sprite Editor window. For example, if you select Delete Existing, it replaces any Sprites that exist (with the same name) with new Sprites; Smart will try to create new Sprites while at the same time adjusting existing ones, and Safe will add new Sprites without changing any that currently exist. The Grid is better for when you have Sprites that are evenly distributed, such as frame-by-frame animations. In these cases, it is not recommended to use Automatic because the size differences between each Sprite may cause unintended effects in terms of how they appear within the game, such as the Pivot being in the wrong location, resulting in an inaccurate animation. An example of the Grid menu is shown in the following screenshot. Pixel Size sets the size of the Grid in the unit of Pixels. This number will be determined based on the size of your Sprite Sheet and distribution of Sprites: Sprite Packer Using the Sprite Packer, you can combine multiple elements such as large sets of Sprites into a single Texture known as an Atlas. However, before using it, we must first make sure that it is enabled within Unity. To do this, go to Edit | Project Settings | Editor. Once you have done this, look at the Inspector; you can change the Sprite Packer from disabled to Always Enabled or vice versa. You can see an example of this in the following screenshot. By selecting Always Enabled. The Sprite Packer will always be enabled whenever you start a new project. That way, you will not need to worry about enabling it again: One of the benefits of using this is that it can boost the performance of your game by reducing the number of Draw Calls each frame. This is because a significant portion of a Sprite Texture will often be taken up by the empty space between the graphic elements. As a result, it wastes video memory at runtime because it needs to compute this empty space even if there is nothing there. By keeping this in mind, when you are creating your own Sprites, try to pack graphics from several Sprite Textures together and as close as possible to one another within an Atlas. Lastly, keep in mind that depending on the sizes of your Sprites, an Atlas should not be larger than 2048 x 2048 or 211 (or at least, this guarantees compatibility with many devices). Unity handles the generation and use of Sprite Atlas Textures behind the scenes so that the user does not need to do any manual assignment. The Atlas can optionally be packed on entering Play mode or during a build, and the graphics for a sprite object will be obtained from the Atlas once it is generated. Users are required to specify a Packing Tag in the Texture Importer to enable packing for Sprites of that Texture. To use the Sprite packer, simply go to the top navigation menu and select Window | Sprite Packer. Once you have done this, it will open the Sprite Packer. Sprite Creator is your friend when you have no assets While we have Sprites, in this case, you might not always have them. If you don't have Sprites, you can always add placeholders or images in the place of where they are likely to be. This is a useful thing to use when you're prototyping an idea and you need to get functionality working before your images are ready to go. Using the Sprite Creator is quite simple. We can create a placeholder Sprite by doing the following: First, select Assets | Create | Sprites. Next, select the placeholder Sprite you want to make, like in the following screenshot. Unity offers only six different placeholder Sprites: Square, Circle, Triangle, Diamond, Hexagon, and Polygon. Before creating the Sprite, it is important to make sure that you select the folder that you want the Sprite to be created in. This just saves time later from having to move it to the correct folder. This is because, when creating a Sprite with the Sprite Creator, it will automatically place it in the Asset folder that you currently have open in the Project Window. Lastly, from the list, select the placeholder Sprite that you wish to use: Once you have chosen your Sprite, it will appear as a white shape. The Texture created by Unity will use the .png file format and contain default image data of 4x4 white pixels. At this stage, the new Sprite will have a default name based on its shapes, such as Square or Circle. You have the option to rename it, but if you don't change it, don't worry, as each additional Sprite that is the same shape will simply have a number following its name. You can, of course, always change the name of the Sprite later by clicking on it in the Asset folder where it is located: Once your new Sprite has been created, simply drag and drop your placeholder Sprite into the Scene view or Hierarchy to start using it in your project. An example of this can be seen in the following screenshot: Once you're done, whether it is a mock-up, prototype, or something else, you may want to change the placeholder Sprite to the actual image. Once you have imported the new image(s), simply do the following: Click on the Sprite within the Scene view so that it is selected. Now, in the Inspector, locate Sprite Renderer Component. An example of this is shown in the following screenshot: Now, where it says Sprite, click on the small circle located next to the Sprite name, in this case, Hexagon. This is highlighted in the following screenshot: Now, a small window will be displayed, like in the following screenshot: The Sprite Creator makes 4x4 white PNG outline Textures, which is a power of 2-sized Texture that is actually generated by an algorithm. Setting up the Angel Cakes project Now we're going to discuss how to set up our first project! For the rest, we're going to discuss how to import the assets for the Angel Cakes project into Unity and get the project ready to go. We'll cover the process for importing and setting up while getting you familiar with 2D assets. To begin, let's get the Angel Cakes asset pack, which is featured in the following screenshot: To download the assets, simply visit www.player26.com/product/Angelcakes and download the .zip file. Once you have finished downloading it, simply unzip the file with a program such as WinRAR. Folder setup You need to make sure that you have some foundational folders created to use with your project. To briefly recap, have a look at the following screenshot. Remember that the Assets folder is always the root or parent folder for the project files: Importing assets into the engine With your folders set up, we now begin to import some images for our project: the background, the player, an enemy, player collision (wall, objects), and collectables (Angel Cakes, health, and bonuses). Importing the assets into Unity is easy. First, click on the folder that you want the Sprites to be imported into, inside the Project window; for this project, we will use the folder titled Sprites Next, in the top menu, click Assets | Import New Assets and navigate to the folder that they are located in Once you have found them, select them and then click Import Once they are imported, they will appear in the folder, like in the following screenshot: Configuring assets for the game The assets used in this game do not need much configuring, in comparison to the ones that we will use later. Once you have imported the two Sprites into Unity, do the following: Select each one within the Project window. Now, in the Inspector, change the Sprite Mode to Multiple. This is because we have multiple images of each Texture. One is an Atlas (the environmental objects) and one is a Sprite Sheet (character animations). Once you have done this, click Apply: Once you have changed the Sprite Mode to Multiple, click Sprite Editor. Now you should see something like the following screenshot: First, click on Slice and select Grid By Cell Size Next, in Pixel Size, change the values of X and Y to 50, like in the following screenshot, then click Slice: Now, if you hold down Ctrl (or command on a Mac) you will see all the freshly cut slices, like in the following screenshot: If you click on each slice, you will notice that a Sprite information box will appear, like in the following screenshot: In this information box, you can rename the Sprite to whatever you would like. Each Sprite has been given a number so that you can understand the corresponding name conventions that are described following screenshot: For this project, we will call each Sprite set the following: Numbers 1-6: ACSpriteChar1...2...3...4... Numbers 7 - 12: ACSpriteCharEvo1...2...3...4... Numbers 13 - 18: ACSpriteCharEnemie1...2...3...4... Number 19: Delete Once you have done this, you can now see all your Sprites within the Project window. To do this, simply click on the triangle that is highlighted in the following screenshot: Once you have clicked this, it will expand, revealing all of your Sprites and their names, like in the following screenshot: There are many things that we will now be able to do with these images, such as animations. The next thing that we need to do now is slice up the environment Atlas. Locate the Sprite file within the Project window and open it up in the Sprite Editor. Remember that you need to change the Sprite type to Multiple in the Inspector, otherwise you will not be able to edit the Sprite. Once you have it in the Sprite Editor, it should look something like the following: This time, instead of selecting the Slice type Grid By Cell Size, we will do it manually. This is because if we choose to do it via the type Automatic, we will find that there are other slices, like those on the clouds on the right of the following screenshot. This can be tedious when there are lots of little parts of a single Sprite, such as the Clouds: So, for now, manually drag and click around each of the Sprites, making sure that you get as close to the edges as possible. You may find that you will need to zoom in on some parts (by using the mouse scroll wheel), like the Angel Cakes. Also, the options in the top-right corner might help you by filtering the image (for example, black and white). As you begin refining the bounding box, you will feel the outline pull or snap toward the edges of the Sprite; this helps you to get as close as possible to the edges, therefore creating more efficient Sprites. Don't forget to name the Sprites either! For this project, we will call each Sprite set the following: ACSpriteEnviroBlock ACSpriteMenuBlock ACSpriteBonus ACSpriteHealth ACSpriteCake ACSpriteBackground ACSpriteCloud1...2...3...and so on To give you a better idea where each Sprite is located, have a look at the following screenshot. The Sprites are numbered so that you can easily locate them. Once you have done this, click on Apply in the top-right corner of the Sprite Editor. As a result, you should be able to see all the Sprites in the Project window by clicking on the triangle. It should look like the following screenshot: 9-slicing Sprites A nice little feature of Unity that allows you to scale elements such as Sprites without distortion is 9-slicing. Essentially, what 9-slicing does is allow you to reuse an image at various sizes without needing to prepare multiple Assets. As the name suggests, it involves splitting the image into nine segments. An example of this splitting is shown in the following screenshot: The following four points describe what will happen if you change the dimensions of the preceding image: If you change the four corners (A, C, G, and I), they will not change in size If you move sections B and H, they will stretch or tile horizontally If you move sections D and F, they will stretch or tile vertically If you move section E, the image will stretch or tile both horizontally and vertically You can see these four points illustrated in the following screenshot: By using 9-slicing, you can re-size the Sprite in different ways and keep the proportions. This is particularly useful for creating the walls within our environment that will create obstacles for our little Angel and enemies to navigate around. We will need to do this for our ACSpriteEnviroBlock so that we can place it within our level for the player to navigate around. To do this, we need to make sure that the Sprite that we have created has been set up properly. First, you need to make sure the Mesh Type is set to Full Rect. To do this, select the Angel_Cake_Sprite_Atlas (contained in Project window | Asset | Sprites), then head to the Inspector and change Mesh Type from Tight to Full Rect, like in the following screenshot: Now we need to define the borders of the Sprite. To do this, perform the following steps: First, select the Sprite (Angel_Cake_Sprite_Atlas). Next, in the Inspector, click the Sprite Editor button. Now, click on the Sprite that you want to apply the 9-slicing to. In our case, this will be the ACSpriteEnviroBlock, like in the following screenshot: Looking at the Sprite information box in the bottom-right corner, we need to adjust the values for the Borders of the Sprite. For this Sprite, we will use the value of 20 for L, R, T, and B (left, right, top, and bottom, respectively): In some cases, you might need to tweak the position of the borders; you can do this by clicking and dragging the green dots located at the intersections of each border (top, bottom, and sides). You can see this in the following screenshot: To test your 9-sliced Sprite, drag it from the Project window into the Scene, like in the following screenshot: Next, in the Inspector, change the Draw Mode from Simple to Sliced, like in the following screenshot: Now you can resize the ACSpriteEnviroBlock without it deforming. Give it a go! You should have something like the variations in the following screenshot:   You will notice that it isn't quite like the Sprite. This is okay, we can adjust this setting in the Inspector. Simply click on the Atlas Texture in the Project window and, in the Inspector, change the value of Pixels Per Unit to 250: Click Apply, then click and drag another ACSpriteEnviroBlock onto the Scene and try to resize it. You will end up with something like the following screenshot: As you can see, there is a little distortion. This just means that you will need to edit the Borders inside the Sprite Editor until you get the location of them correct. For now, tinker with the locations of the borders. To summarize, we have shown how to work with the Unity 2017 assets, and how you work and configure sprites for your 2D game projects effectively. If you have liked this article, then don't forget to check out the complete book Unity 2017 2D Game Development Projects   by Lauren S. Ferro & Francesco Sapio on the Packt store. Working with Unity Variables to script powerful Unity 2017 games Build an ARCore app with Unity from scratch Unity announces a new automotive division and two-day Unity AutoTech Summit

The Search pages in the Database or Table view are intended for single-table lookups. This article by Marc Delisle, covers the multi-table Query by example (QBE) feature available in the Database view. Many phpMyAdmin users work in the Table view, table-by-table, and thus tend to overlook the multi-table query generator, which is a wonderful feature for fine-tuning queries. The query generator is useful not only in multi-table situations but also for a single table. It enables us to specify multiple criteria for a column, a feature that the Search page in the Table view does not possess. The examples in this article assumes that a multi-user installation of the linked-tables infrastructure has been made and that the book-copy table created during an exercise in the article on Table and Database Operations in PHP is still there in the marc_book database. To access the code used in this article Click Here. To open the page for this feature, we go to the Database view for a specific database (the query generator supports working on only one database at a time) and click on Query. The screenshot overleaf shows the initial QBE page. It contains the following elements: Criteria columns An interface to add criteria rows An interface to add criteria columns A table selector The query area Buttons to update or to execute the query Choosing Tables The initial selection includes all the tables. In this example, we assume that the linked-table infrastructure has been installed into the marc_book database. Consequently, the Field selector contains a great number of fields. For our example, we will work only with the author and book tables: We then click Update Query. This refreshes the screen and reduces the number of fields available in the Field selector. We can always change the table choice later, using our browser's mechanism for multiple choices in drop-down menus (usually, control-click). Column Criteria Three criteria columns are provided by default. This section discusses the options we have for editing their criteria. These include options for selecting fields, sorting individual columns, entering conditions for individual columns, and so on. Field Selector: Single-Column or All Columns The Field selector contains all individual columns for the selected tables, plus a special choice ending with an asterisk (*) for each table, which means all the fields are selected: To display all the fields in the author table, we choose `author`.* and check the Show checkbox, without entering anything in the Sort and Criteria boxes. In our case, we select `author`.`name`, because we want to enter some criteria for the author's name. Sorts For each selected individual column, we can specify a sort (in Ascending or Descending order) or let this line remain intact (meaning no sort). If we choose more than one sorted column, the sort will be done with a priority from left to right. When we ask for a column to be sorted, we normally check the Show checkbox, but this is not necessary because we might want to do just the sorting operation without displaying this column. Showing a Column We check the Show checkbox so that we can see the column in the results. Sometimes, we may just want to apply a criterion on a column and not include it in the resulting page. Here, we add the phone column, ask for a sort on it, and choose to show both the name and phone number. We also ask for a sort on the name in ascending order. The sort will be done first by name, and then by phone number, if the names are identical. This is because the name is in a column criterion to the left of the phone column, and thus has a higher priority:

In this article by Asif Abbasi author of the book Learning Apache Spark 2.0, we will understand Spark RDD along with that we will learn, how to construct RDDs, Operations on RDDs, Passing functions to Spark in Scala, Java, and Python and Transformations such as map, filter, flatMap, and sample. (For more resources related to this topic, see here.) What is an RDD? What’s in a name might be true for a rose, but perhaps not for an Resilient Distributed Datasets (RDD), and in essence describes what an RDD is. They are basically datasets, which are distributed across a cluster (remember Spark framework is inherently based on an MPP architecture), and provide resilience (automatic failover) by nature. Before we go into any further detail, let’s try to understand this a little bit, and again we are trying to be as abstract as possible. Let us assume that you have a sensor data from aircraft sensors and you want to analyze the data irrespective of its size and locality. For example, an Airbus A350 has roughly 6000 sensors across the entire plane and generates 2.5 TB data per day, while the newer model expected to launch in 2020 will generate roughly 7.5 TB per day. From a data engineering point of view, it might be important to understand the data pipeline, but from an analyst and a data scientist point of view, your major concern is to analyze the data irrespective of the size and number of nodes across which it has been stored. This is where the neatness of the RDD concept comes into play, where the sensor data can be encapsulated as an RDD concept, and any transformation/action that you perform on the RDD applies across the entire dataset. Six month's worth of dataset for an A350 would be approximately 450 TBs of data, and would need to sit across multiple machines. For the sake of discussion, we assume that you are working on a cluster of four worker machines. Your data would be partitioned across the workers as follows: Figure 2-1: RDD split across a cluster The figure basically explains that an RDD is a distributed collection of the data, and the framework distributes the data across the cluster. Data distribution across a set of machines brings its own set of nuisances including recovering from node failures. RDD’s are resilient as they can be recomputed from the RDD lineage graph, which is basically a graph of the entire parent RDDs of the RDD. In addition to the resilience, distribution, and representing a data set, an RDD has various other distinguishing qualities: In Memory: An RDD is a memory resident collection of objects. We’ll look at options where an RDD can be stored in memory, on disk, or both. However, the execution speed of Spark stems from the fact that the data is in memory, and is not fetched from disk for each operation. Partitioned: A partition is a division of a logical dataset or constituent elements into independent parts. Partitioning is a defacto performance optimization technique in distributed systems to achieve minimal network traffic, a killer for high performance workloads. The objective of partitioning in key-value oriented data is to collocate similar range of keys and in effect minimize shuffling. Data inside RDD is split into partitions and across various nodes of the cluster. Typed: Data in an RDD is strongly typed. When you create an RDD, all the elements are typed depending on the data type. Lazy evaluation: The transformations in Spark are lazy, which means data inside RDD is not available until you perform an action. You can, however, make the data available at any time using a count() action on the RDD. We’ll discuss this later and the benefits associated with it. Immutable: An RDD once created cannot be changed. It can, however, be transformed into a new RDD by performing a set of transformations on it. Parallel: An RDD is operated on in parallel. Since the data is spread across a cluster in various partitions, each partition is operated on in parallel. Cacheable: Since RDD’s are lazily evaluated, any action on an RDD will cause the RDD to revaluate all transformations that led to the creation of RDD. This is generally not a desirable behavior on large datasets, and hence Spark allows the option to persist the data on memory or disk. A typical Spark program flow with an RDD includes: Creation of an RDD from a data source. A set of transformations, for example, filter, map, join, and so on. Persisting the RDD to avoid re-execution. Calling actions on the RDD to start performing parallel operations across the cluster. This is depicted in the following figure: Figure 2-2: Typical Spark RDD flow Operations on RDD Two major operation types can be performed on an RDD. They are called: Transformations Actions Transformations Transformations are operations that create a new dataset, as RDDs are immutable. They are used to transform data from one to another, which could result in amplification of the data, reduction of the data, or a totally different shape altogether. These operations do not return any value back to the driver program, and hence lazily evaluated, which is one of the main benefits of Spark. An example of a transformation would be a map function that will pass through each element of the RDD and return a totally new RDD representing the results of application of the function on the original dataset. Actions Actions are operations that return a value to the driver program. As previously discussed, all transformations in Spark are lazy, which essentially means that Spark remembers all the transformations carried out on an RDD, and applies them in the most optimal fashion when an action is called. For example, you might have a 1 TB dataset, which you pass through a set of map functions by applying various transformations. Finally, you apply the reduce action on the dataset. Apache Spark will return only a final dataset, which might be few MBs rather than the entire 1 TB dataset of mapped intermediate result. You should, however, remember to persist intermediate results otherwise Spark will recompute the entire RDD graph each time an Action is called. The persist() method on an RDD should help you avoid recomputation and saving intermediate results. We’ll look at this in more detail later. Let’s illustrate the work of transformations and actions by a simple example. In this specific example, we’ll be using flatmap() transformations and a count action. We’ll use the README.md file from the local filesystem as an example. We’ll give a line-by-line explanation of the Scala example, and then provide code for Python and Java. As always, you must try this example with your own piece of text and investigate the results: //Loading the README.md file val dataFile = sc.textFile(“README.md”) Now that the data has been loaded, we’ll need to run a transformation. Since we know that each line of the text is loaded as a separate element, we’ll need to run a flatMap transformation and separate out individual words as separate elements, for which we’ll use the split function and use space as a delimiter: //Separate out a list of words from individual RDD elements val words = dataFile.flatMap(line => line.split(“ “)) Remember that until this point, while you seem to have applied a transformation function, nothing has been executed and all the transformations have been added to the logical plan. Also note that the transformation function returns a new RDD. We can then call the count() action on the words RDD, to perform the computation, which then results in fetching of data from the file to create an RDD, before applying the transformation function specified. You might note that we have actually passed a function to Spark: //Separate out a list of words from individual RDD elements Words.count() Upon calling the count() action the RDD is evaluated, and the results are sent back to the driver program. This is very neat and especially useful during big data applications. If you are Python savvy, you may want to run the following code in PySpark. You should note that lambda functions are passed to the Spark framework: //Loading data file, applying transformations and action dataFile = sc.textFile("README.md") words = dataFile.flatMap(lambda line: line.split(" ")) words.count() Programming the same functionality in Java is also quite straight forward and will look pretty similar to the program in Scala: JavaRDD<String> lines = sc.textFile("README.md"); JavaRDD<String> words = lines.map(line -> line.split(“ “)); int wordCount = words.count(); This might look like a simple program, but behind the scenes it is taking the line.split(“ ”) function and applying it to all the partitions in the cluster in parallel. The framework provides this simplicity and does all the background work of coordination to schedule it across the cluster, and get the results back. Passing functions to Spark (Scala) As you have seen in the previous example, passing functions is a critical functionality provided by Spark. From a user’s point of view you would pass the function in your driver program, and Spark would figure out the location of the data partitions across the cluster memory, running it in parallel. The exact syntax of passing functions differs by the programming language. Since Spark has been written in Scala, we’ll discuss Scala first. In Scala, the recommended ways to pass functions to the Spark framework are as follows: Anonymous functions Static singleton methods Anonymous functions Anonymous functions are used for short pieces of code. They are also referred to as lambda expressions, and are a cool and elegant feature of the programming language. The reason they are called anonymous functions is because you can give any name to the input argument and the result would be the same. For example, the following code examples would produce the same output: val words = dataFile.map(line => line.split(“ “)) val words = dataFile.map(anyline => anyline.split(“ “)) val words = dataFile.map(_.split(“ “)) Figure 2-11: Passing anonymous functions to Spark in Scala Static singleton functions While anonymous functions are really helpful for short snippets of code, they are not very helpful when you want to request the framework for a complex data manipulation. Static singleton functions come to the rescue with their own nuances, which we will discuss in this section. In software engineering, the Singleton pattern is a design pattern that restricts instantiation of a class to one object. This is useful when exactly one object is needed to coordinate actions across the system. Static methods belong to the class and not an instance of it. They usually take input from the parameters, perform actions on it, and return the result. Figure 2-12: Passing static singleton functions to Spark in Scala Static singleton is the preferred way to pass functions, as technically you can create a class and call a method in the class instance. For example: class UtilFunctions{ def split(inputParam: String): Array[String] = {inputParam.split(“ “)} def operate(rdd: RDD[String]): RDD[String] ={ rdd.map(split)} } You can send a method in a class, but that has performance implications as the entire object would be sent along the method. Passing functions to Spark (Java) In Java, to create a function you will have to implement the interfaces available in the org.apache.spark.api.java function package. There are two popular ways to create such functions: Implement the interface in your own class, and pass the instance to Spark. Starting Java 8, you can use lambda expressions to pass off the functions to the Spark framework. Let’s reimplement the preceding word count examples in Java: Figure 2-13: Code example of Java implementation of word count (inline functions) If you belong to a group of programmers who feel that writing inline functions makes the code complex and unreadable (a lot of people do agree to that assertion), you may want to create separate functions and call them as follows: Figure 2-14: Code example of Java implementation of word count Passing functions to Spark (Python) Python provides a simple way to pass functions to Spark. The Spark programming guide available at spark.apache.org suggests there are three recommended ways to do this: Lambda expressions: The ideal way for short functions that can be written inside a single expression Local defs inside the function calling into Spark for longer code Top-level functions in a module While we have already looked at the lambda functions in some of the previous examples, let’s look at local definitions of the functions. Our example stays the same, which is we are trying to count the total number of words in a text file in Spark: def splitter(lineOfText): words = lineOfText.split(" ") return len(words) def aggregate(numWordsLine1, numWordsLineNext): totalWords = numWordsLine1 + numWordsLineNext return totalWords Let’s see the working code example: Figure 2-15: Code example of Python word count (local definition of functions) Here’s another way to implement this by defining the functions as a part of a UtilFunctions class, and referencing them within your map and reduce functions: Figure 2-16: Code example of Python word count (Utility class) You may want to be a bit cheeky here and try to add a countWords() to the UtilFunctions, so that it takes an RDD as input, and returns the total number of words. This method has potential performance implications as the whole object will need to be sent to the cluster. Let’s see how this can be implemented and the results in the following screenshot: Figure 2-17: Code example of Python word count (Utility class - 2) This can be avoided by making a copy of the referenced data field in a local object, rather than accessing it externally. Now that we have had a look at how to pass functions to Spark, and have already looked at some of the transformations and actions in the previous examples, including map, flatMap, and reduce, let’s look at the most common transformations and actions used in Spark. The list is not exhaustive, and you can find more examples in the Apache Spark documentation in the programming guide. If you would like to get a comprehensive list of all the available functions, you might want to check the following API docs:   RDD PairRDD Scala http://bit.ly/2bfyoTo http://bit.ly/2bfzgah Python http://bit.ly/2bfyURl N/A Java http://bit.ly/2bfyRov http://bit.ly/2bfyOsH R http://bit.ly/2bfyrOZ N/A Table 2.1 – RDD and PairRDD API references Transformations The following table shows the most common transformations: map(func) coalesce(numPartitions) filter(func) repartition(numPartitions) flatMap(func) repartitionAndSortWithinPartitions(partitioner) mapPartitions(func) join(otherDataset, [numTasks]) mapPartitionsWithIndex(func) cogroup(otherDataset, [numTasks]) sample(withReplacement, fraction, seed) cartesian(otherDataset) Map(func) The map transformation is the most commonly used and the simplest of transformations on an RDD. The map transformation applies the function passed in the arguments to each of the elements of the source RDD. In the previous examples, we have seen the usage of map() transformation where we have passed the split() function to the input RDD. Figure 2-18: Operation of a map() function We’ll not give examples of map() functions as we have already seen plenty of examples of map functions previously. Filter (func) Filter, as the name implies, filters the input RDD, and creates a new dataset that satisfies the predicate passed as arguments. Example 2-1: Scala filtering example: val dataFile = sc.textFile(“README.md”) val linesWithApache = dataFile.filter(line => line.contains(“Apache”)) Example 2-2: Python filtering example: dataFile = sc.textFile(“README.md”) linesWithApache = dataFile.filter(lambda line: “Apache” in line) Example 2-3: Java filtering example: JavaRDD<String> dataFile = sc.textFile(“README.md”) JavaRDD<String> linesWithApache = dataFile.filter(line -> line.contains(“Apache”)); flatMap(func) The flatMap transformation is similar to map, but it offers a bit more flexibility. From the perspective of similarity to a map function, it operates on all the elements of the RDD, but the flexibility stems from its ability to handle functions that return a sequence rather than a single item. As you saw in the preceding examples, we had used flatMap to flatten the result of the split(“”) function, which returns a flattened structure rather than an RDD of string arrays. Figure 2-19: Operational details of the flatMap() transformation Let’s look at the flatMap example in Scala. Example 2-4: The flatmap() example in Scala: val favMovies = sc.parallelize(List("Pulp Fiction","Requiem for a dream","A clockwork Orange")); movies.flatMap(movieTitle=>movieTitle.split(" ")).collect() A flatMap in Python API would produce similar results. Example 2-5: The flatmap() example in Python: movies = sc.parallelize(["Pulp Fiction","Requiem for a dream","A clockwork Orange"]) movies.flatMap(lambda movieTitle: movieTitle.split(" ")).collect() The flatMap example in Java is a bit long-winded, but it essentially produces the same results. Example 2-6: The flatmap() example in Java: JavaRDD<String> movies = sc.parallelize (Arrays.asList("Pulp Fiction","Requiem for a dream" ,"A clockwork Orange") ); JavaRDD<String> movieName = movies.flatMap( new FlatMapFunction<String,String>(){ public Iterator<String> call(String movie){ return Arrays.asList(movie.split(" ")) .iterator(); } } ); Sample(withReplacement, fraction, seed) Sampling is an important component of any data analysis and it can have a significant impact on the quality of your results/findings. Spark provides an easy way to sample RDD’s for your calculations, if you would prefer to quickly test your hypothesis on a subset of data before running it on a full dataset. But here is a quick overview of the parameters that are passed onto the method: withReplacement: Is a Boolean (True/False), and it indicates if elements can be sampled multiple times (replaced when sampled out). Sampling with replacement means that the two sample values are independent. In practical terms this means that if we draw two samples with replacement, what we get on the first one doesn’t affect what we get on the second draw, and hence the covariance between the two samples is zero. If we are sampling without replacement, the two samples aren’t independent. Practically this means what we got on the first draw affects what we get on the second one and hence the covariance between the two isn’t zero. fraction: Fraction indicates the expected size of the sample as a fraction of the RDD’s size. The fraction must be between 0 and 1. For example, if you want to draw a 5% sample, you can choose 0.05 as a fraction. seed: The seed used for the random number generator. Let’s look at the sampling example in Scala. Example 2-7: The sample() example in Scala: val data = sc.parallelize( List(1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20)); data.sample(true,0.1,12345).collect() The sampling example in Python looks similar to the one in Scala. Example 2-8: The sample() example in Python: data = sc.parallelize( [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]) data.sample(1,0.1,12345).collect() In Java, our sampling example returns an RDD of integers. Example 2-9: The sample() example in Java: JavaRDD<Integer> nums = sc.parallelize(Arrays.asList( 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20)); nums.sample(true,0.1,12345).collect(); References https://spark.apache.org/docs/latest/programming-guide.html http://www.purplemath.com/modules/numbprop.htm Summary We have gone through the concept of creating an RDD, to manipulating data within the RDD. We’ve looked at the transformations and actions available to an RDD, and walked you through various code examples to explain the differences between transformations and actions Resources for Article: Further resources on this subject: Getting Started with Apache Spark [article] Getting Started with Apache Spark DataFrames [article] Sabermetrics with Apache Spark [article]

In this article by Ankit Jain, the author of the book Mastering Apache Storm,This section we will seehow you how we can start the Storm UI daemon. However, before starting the Storm UI daemon, we assume that you have a running Storm cluster. The Storm cluster deployment steps are mentioned in previous step. NNow, go to the Storm home directory (cd $STORM_HOME) at the Leader Nimbus machine and run the following command to start the Storm UI daemon: $> cd $STORM_HOME $> bin/storm ui & (For more resources related to this topic, see here.) By default, the Storm UI starts on the 8080 port of the machine where it is started. Now, we will browse to the http://nimbus-node:8080 page to view the Storm UI, where Nnimbus-node is the IP address or hostname of the Nimbus machine. The following is a screenshot of the Storm home page: Insert Image B06182_Article02_0103.png Cluster Summary Section This portion of the Storm UI shows the version of Storm deployed in a cluster, uptime of the Nnimbus nodes, number of free worker slots, number of used worker slots, and so on. While submitting a topology to the cluster, the user first needs to make sure that the value of the Free slots column should not be zero; otherwise, the topology doesn't get any worker for processing and will wait in the queue till a worker becomes free. Nimbus Summary section This portion of the Storm UI shows the number of Nimbus processes are running in Storm Cluster. The section also shows Status of Nimbus nodes. A node with Status "Leader" is an Active master while the node with Status is "Not a Leader" is a Passive master. Supervisor Summary section This portion of the Storm UI shows the list of supervisor nodes running in the cluster along with their Id, Host, Uptime, Slots, and Used slots columns. Nimbus Configuration section This portion of the Storm UI shows the configuration of the Nimbus node. Some of the important properties are: supervisor.slots.ports storm.zookeeper.port storm.zookeeper.servers storm.zookeeper.retry.interval worker.childopts supervisor.childopts The definition of each of this property are covert in chapter 3. Following is the screenshots of Nimbus Configuration: Topology Summary section This portion of the Storm UI shows the list of topologies running in the Storm cluster along with their ID, number of workers assigned to the topology, number of executors, number of tasks, uptime, and so on. Let's deploy the sample topology (if not running already) in a remote Storm cluster by running the following command: $> cd $STORM_HOME $> bin/storm jar ~/storm_example-0.0.1-SNAPSHOT-jar-with-dependencies.jar com.stormadvance.storm_example.SampleStormClusterTopology storm_example We have created SampleStormClusterTopology topology by defining three worker processes, two executors for SampleSpout, and four executors for SampleBolt. The information about the worker, executor and task is mentioned in next chapter. After submitting SampleStormClusterTopology on the Storm cluster, the user has to refresh the Storm home page. The following screenshot shows that the row is added for SampleStormClusterTopology in the Topology summary section. The topology section contains the name of the topology, unique ID of the topology, status of the topology, uptime, number of workers assigned to the topology, and so on. The possible values of status fields are ACTIVE, KILLED, and INACTIVE. Let's click on SampleStormClusterTopology to view its detailed statistics. I am attaching there are two screenshots for the same. The first one content the information about the number of numberworkers, executors, and tasks assigned to SampleStormClusterTopology topology The next screenshot image contains information about spout and bolts— The number of executors and tasks assigned to each spout and bolt: The information showed in the previous screenshots are: Topology stats: This section will give the information about the number of tuples emitted, transferred, and acknowledged, the capacity latency, and so on, within the window of 10 minutes, 3 hours, 1 day, and since the start of the topology. Spouts (All time): This section shows the statistics of all the spouts running inside a topology.The detailed information of Spout stats is mentioned in chapter 3. Bolts (All time): This section shows the statistics of all the bolts running inside a topology. The detailed information of Bolt stats is mentioned in chapter 3. Topology actions: This section allows us to perform activate, deactivate, rebalance, kill, and other operation on topology's directly through the Storm UI. Deactivate: Click on deactivate to deactivate the topology. Once the topology is deactivate, the spout stopped emitting tuples and the status of topology changed to INACTIVE on storm UI  Deactivating the topology does not free the Storm resource. Activate: Click on Activate button to activate the topology. Once the topology is activate, the spout again started emitting tuples. Kill: Click on Kill button to destroy/kill the topology. Once the topology is killed, it will free all the Storm resource allotted to this topology. While killing the topology, Storm will first deactivate the spouts and wait for the kill time mentioned on the alerts box, so the bolts have a chance to finish the processing of the tuples emitted by spouts before the kill command. The following screenshot shows how we can kill the topology through the Storm UI: Let's go to the Storm UI's home page to check the status of SampleStormClusterToplogy, as shown in the following screenshot: Summary We have seen how to start Storm UI daemon and we have also seen Storm home page with its sections Resources for Article:   Further resources on this subject: [article] [article] [article]

Create Models or Use a Library? There are two possibilities when working with furniture. We can create new furniture, or use pre-made models from a library. The question is: when must we use each type? Some people say that using a pre-made model is not very professional thing but what they forget to say is that most projects have a tight deadline, and we need a quick modeling process to be ready on time. So, what's most important for professionals? Getting things done, or telling the client that all the models were created just for his project? Of course, the deadline is the most important, and your clients normally won't mind if you use pre-made models. Probably they won't even notice. So don't be ashamed to use pre-made models they won't make your projects any less professional. It's even recommended to use these models to speed-up the process, and allow you to spend more time on lighting or texturing. Is there any situation that demands the creation of a furniture model from scratch? Well, there are some. First, if you can't find the model in any library that you know, then it's going to be necessary to create it from scratch. If you are working with an architect who designs the spaces and furniture as well, you will probably have to model the furniture too, since it won't be available at any public library. Any project that deals with customized furniture will require that we work on the modeling for the furniture. Create your own libraryA good practice for anyone doing architectural visualization is to collect a lot of 3D models from public libraries for use in future projects. Keep these models for later, but don't forget to check if the author has released the models with no restrictions for commercial use. Otherwise, you must get their permission to use them. If you want to create your library, with no restrictions, why not create your own models? This could be a good exercise: take a few examples, and start creating some furniture. With time, you will have a good number of models. How to Get Started? In most cases, we have to get used to all that furniture modeling. We will have to start from scratch, with no blueprints available. The only references that we will have would be the photos, either provided by our clients, or provided from some web resources. If you have the time, visit a real store, and take some pictures and measures on your own. Sometimes, these stores will give you fliers and brochures, especially if you work with architecture. With time, you will get a lot of good reference material, and some of them come with measurements. But, if you don't know where to get started, let me point out some great web resources: http://www.e-interiors.net http://resources.blogoscopia.com http://blender-archi.tuxfamily.org/Models http://www.katorlegaz.com/ http://sketchup.google.com/3dwarehouse The first link has a lot of reference images classified by furniture type and designer. And sometimes, they even provide free 3D models. Most models there are saved in DXF, or 3DS file formats. Appending Models Before we start to model, let's see how we can import a model form an external library into Blender. The process is very simple, and what we have to do is to use the File menu, and access the Append or Link option. There is a shortcut for that too - just press SHIFT+F1 to call the same function. With this option, we have to select file that is already in the Blender file format. This option won't import files in other formats. When we select a file, a list of elements available in that particular file will be displayed, for us to select what we want to import. In most cases, the models will be stored under Object. When we click the Object option, all of the objects available in the file will be listed. If you know the name of the object that you want to import, just select the name, and click Load Library. The object will be loaded into our scene. Here, we have two options to handle this object: Append or Link: Append: If we choose this option, the object will be merged into ourcurrent scene. Link: With this option, an external link to the object file will be created. Any modifications to the original file will be reflected in our current scene. What is the best method to use? It will depend on whether we are willing to track all modifications applied to our furniture models. Using the Link method is a great way of keeping the furniture updated, because every modification at the original file is reflected immediately in the scene in which this model is placed. However, we will have to take the original file with the scene file every time we need to put our scene on another computer. They always have to go together. But if you choose to use the Append option, things will be a bit simpler, because the object will be incorporated into the scene file. We won't have to be worried about moving the furniture file along with the scene. Always use the Append option when you want to use furniture, or any other model, saved in another Blender file. To use a furniture model saved in another file, with a type other than “.blend”, we have to use the Import option. Importing Models To import a model, the process is very simple. We must use the File menu and select, Import. Then we have to select the proper file type from the list. The best file type, and the most common for furniture blocks, is the 3Ds file format, which belongs to the old 3D Studio application. There are some other good formats that work well with Blender, such as OBJ and LWO. The 3Ds file format can store lights, and it works well with Blender. The only thing we have to take extra care about is that most models imported come with triangular faces, which are a bit harder to edit. But, if you don't need to make any modifications to the model, this won't be a problem. Append or Import?Just to make things clearer, if you download a furniture model from a web site, and it's saved in the Blender native file format (.blend), you should append the model. If you download or get a furniture model on any file format other than “.blend”, you will have to import it. Since most models aren't saved in the Blender native file format, we can safely say that almost all furniture models that you find will require an import action to be placed in your scenes. Modeling a Chair Let's start with something simple, such as a chair. Even for a simple model, it will help us deal with smaller dimensions and details. Here is an image of the model: What's the main objective of this modeling? We have to create this chair, with the minimum use of faces and vertices. A good amount of detail can be left for textures, and it's always a good choice to use a lower number of vertices and faces in a model. If you consider one model, it won't matter much. But with a large number of chairs, such as in a theater room, it can make a difference in render time. Let's get started with a simple cube. Select this cube, and change the work mode to Edit. Select all vertices and press the W key. This will open the Specials menu. Choose subdivide, just once, from this menu. This will create new vertices and edges. Once these new vertices have been created, as shown in the image to the left, below, press the A key to remove all of the objects from the selection. Now, select the vertices to the right, using the B key. Remember to change the view mode to Wireframe before using the B key, otherwise, we won't be able to select the vertices behind the visible faces. When these vertices are selected, press the X key and choose Vertices to erase only the selected vertices. Using the CTRL+R key, add a new edge loop to the model, as shown in the following image: The next step is to change the scale of our model. Rotate the view to see the model in perspective view. Select all objects and press the S key, immediately after pressing the Z key. This will make the scale work only in the Z axis. Now, select the vertices identified in the following image and erase them using the X key. Change the selection mode to Edges, and select the edges identified in the following image. With the edges selected, press the E key to extrude them. With the new faces created, we can now add some detail to the model. Select only the top edge of the previously created faces. Move this edge down just a bit. This will add a small declivity to the seat. Now, we can move on to the next extrude, which must be from the selected edges identified in the following image. I'm not using any kind of measure for this example, but if you like to work only with real measurements, remember to hold the CTRL key every time a new extrude or edge is moved. This way, all transformations will use the grid lines. For this model, I'm not using vertex snap. With the new faces created, select just the two edges identified in the following image. Extrude these edges until they reach the other side of the base model. Hold the CTRL key, while you extrude them, to help with the precision. If you already want to remove duplicated vertices, select all objects, and press the W key. Choose Remove doubles to erase any duplicated vertices. Select the edges identified in the image to keep adding more parts to the chair. Extrude the edges three times until you have the same structure showed here. Now, we have to close the top with a face. To do that, we must select all four vertices on the top. When the vertices are selected, press the F key to create a new face. The next step is to select the small side edges to create some detail. Select just one edge, beginning from bottom to top, and move it just a bit. Repeat this operation with the other edges until we get the edges positioned as in the following image. The basic shape of our chair has now been created. Now, we can make some adjustments for improving the overall proportions. Select all edges or vertices on the left side, and move them a bit to the left. This will make the model wider. Did you notice that we have modeled only half a chair? Now we can make the other half, using the Mirror modifier. Add the modifier, and choose the right axis to make a perfect copy. If the center point for the model has been moved, you might need to edit the model to create a perfect mirrored match. Don't worry if you have moved the model by accident - this can happen sometimes. Along with the Mirror modifier, add a Subsurf modifier, too. With the Subsurf modifier, we realize that this model needs a new edge loop on the left side. Just press CTRL+R, and add a new loop, as in the following image.

(For more resources related to this topic, see here.) Mockito is an open source framework for Java that allows you to easily create test doubles (mocks). What makes Mockito so special is that it eliminates the common expect-run-verify pattern (which was present, for example, in EasyMock—please refer to http://monkeyisland.pl/2008/02/24/can-i-test-what-i-want-please for more details) that in effect leads to a lower coupling of the test code to the production code as such. In other words, one does not have to define the expectations of how the mock should behave in order to verify its behavior. That way, the code is clearer and more readable for the user. On one hand, Mockito has a very active group of contributors and is actively maintained. On the other hand, by the time this article is written, the last Mockito release (Version 1.9.5) would have been in October 2012. You may ask yourself the question, "Why should I even bother to use Mockito in the first place?" Out of many, Mockito offers the following key features: There is no expectation phase for Mockito—you can either stub or verify the mock's behavior You are able to mock both interfaces and classes You can produce little boilerplate code while working with Mockito by means of annotations You can easily verify or stub with intuitive argument matchers Before diving into Mockito as such, one has to understand the concept behind System Under Test (SUT) and test doubles. We will base on what Gerard Meszaros has defined in the xUnit Patterns (http://xunitpatterns.com/Mocks,%20Fakes,%20Stubs%20and%20Dummies.html). SUT (http://xunitpatterns.com/SUT.html) describes the system that we are testing. It doesn't have to necessarily signify a class but any part of the application that we are testing or even the whole application as such. As for test doubles (http://www.martinfowler.com/bliki/TestDouble.html), it's an object that is used only for testing purposes, instead of a real object. Let's take a look at different types of test doubles: Dummy: This is an object that is used only for the code to compile—it doesn't have any business logic (for example, an object passed as a parameter to a method) Fake: This is an object that has an implementation but it's not production ready (for example, using an in-memory database instead of communicating with a standalone one) Stub: This is an object that has predefined answers to method executions made during the test Mock: This is an object that has predefined answers to method executions made during the test and has recorded expectations of these executions Spy: These are objects that are similar to stubs, but they additionally record how they were executed (for example, a service that holds a record of the number of sent messages) An additional remark is also related to testing the output of our application. The more decoupled your test code is from your production code, the better since you will have to spend less time (or even none) on modifying your tests after you change the implementation of the code. Coming back to the article's content—this article is all about getting started with Mockito. We will begin with how to add Mockito to your classpath. Then, we'll see a simple setup of tests for both JUnit and TestNG test frameworks. Next, we will check why it is crucial to assert the behavior of the system under test instead of verifying its implementation details. Finally, we will check out some of Mockito's experimental features, adding hints and warnings to the exception messages. The very idea of the following recipes is to prepare your test classes to work with Mockito and to show you how to do this with as little boilerplate code as possible. Due to my fondness of the behavior driven development (http://dannorth.net/introducing-bdd/ first introduced by Dan North), I'm using Mockito's BDDMockito and AssertJ's BDDAssertions static methods to make the code even more readable and intuitive in all the test cases. Also, please read Szczepan Faber's blog (author of Mockito) about the given, when, then separation in your test methods—http://monkeyisland.pl/2009/12/07/given-when-then-forever/—since these are omnipresent throughout the article. I don't want the article to become a duplication of the Mockito documentation, which is of high quality—I would like you to take a look at good tests and get acquainted with Mockito syntax from the beginning. What's more, I've used static imports in the code to make it even more readable, so if you get confused with any of the pieces of code, it would be best to consult the repository and the code as such. Adding Mockito to a project's classpath Adding Mockito to a project's classpath is as simple as adding one of the two jars to your project's classpath: mockito-all: This is a single jar with all dependencies (with the hamcrest and objenesis libraries—as of June 2011). mockito-core: This is only Mockito core (without hamcrest or objenesis). Use this if you want to control which version of hamcrest or objenesis is used. How to do it... If you are using a dependency manager that connects to the Maven Central Repository, then you can get your dependencies as follows (examples of how to add mockito-all to your classpath for Maven and Gradle): For Maven, use the following code: <dependency> <groupId>org.mockito</groupId> <artifactId>mockito-all</artifactId> <version>1.9.5</version> <scope>test</scope> </dependency> For Gradle, use the following code: testCompile "org.mockito:mockito-all:1.9.5" If you are not using any of the dependency managers, you have to either download mockito-all.jar or mockito-core.jar and add it to your classpath manually (you can download the jars from https://code.google.com/p/mockito/downloads/list). Getting started with Mockito for JUnit Before going into details regarding Mockito and JUnit integration, it is worth mentioning a few words about JUnit. JUnit is a testing framework (an implementation of the xUnit famework) that allows you to create repeatable tests in a very readable manner. In fact, JUnit is a port of Smalltalk's SUnit (both the frameworks were originally implemented by Kent Beck). What is important in terms of JUnit and Mockito integration is that under the hood, JUnit uses a test runner to run its tests (from xUnit—test runner is a program that executes the test logic and reports the test results). Mockito has its own test runner implementation that allows you to reduce boilerplate in order to create test doubles (mocks and spies) and to inject them (either via constructors, setters, or reflection) into the defined object. What's more, you can easily create argument captors. All of this is feasible by means of proper annotations as follows: @Mock: This is used for mock creation @Spy: This is used to create a spy instance @InjectMocks: This is used to instantiate the @InjectMock annotated field and inject all the @Mock or @Spy annotated fields into it (if applicable) @Captor: This is used to create an argument captor By default, you should profit from Mockito's annotations to make your code look neat and to reduce the boilerplate code in your application. Getting ready In order to add JUnit to your classpath, if you are using a dependency manager that connects to the Maven Central Repository, then you can get your dependencies as follows (examples for Maven and Gradle): To add JUnit in Maven, use the following code: <dependency> <groupId>junit</groupId> <artifactId>junit</artifactId> <version>4.11</version> <scope>test</scope> </dependency> To add JUnit in Gradle, use the following code: testCompile('junit:junit:4.11') If you are not using any of the dependency managers, you have to download the following jars: junit.jar hamcrest-core.jar Add the downloaded files to your classpath manually (you can download the jars from https://github.com/junit-team/junit/wiki/Download-and-Install). For this recipe, our system under test will be a MeanTaxFactorCalculator class that will call an external service, TaxService, to get the current tax factor for the current user. It's a tax factor and not tax as such since, for simplicity, we will not be using BigDecimals but doubles, and I'd never suggest using doubles to anything related to money, as follows: public class MeanTaxFactorCalculator { private final TaxService taxService; public MeanTaxFactorCalculator(TaxService taxService) { this.taxService = taxService; } public double calculateMeanTaxFactorFor(Person person) { double currentTaxFactor = taxService.getCurrentTaxFactorFor(person); double anotherTaxFactor = taxService.getCurrentTaxFactorFor(person); return (currentTaxFactor + anotherTaxFactor) / 2; } } How to do it... To use Mockito's annotations, you have to perform the following steps: Annotate your test with the @RunWith(MockitoJUnitRunner.class). Annotate the test fields with the @Mock or @Spy annotation to have either a mock or spy object instantiated. Annotate the test fields with the @InjectMocks annotation to first instantiate the @InjectMock annotated field and then inject all the @Mock or @Spy annotated fields into it (if applicable). The following snippet shows the JUnit and Mockito integration in a test class that verifies the SUT's behavior (remember that I'm using BDDMockito.given(...) and AssertJ's BDDAssertions.then(...) static methods: @RunWith(MockitoJUnitRunner.class) public class MeanTaxFactorCalculatorTest { static final double TAX_FACTOR = 10; @Mock TaxService taxService; @InjectMocks MeanTaxFactorCalculator systemUnderTest; @Test public void should_calculate_mean_tax_factor() { // given given(taxService.getCurrentTaxFactorFor(any(Person.class))).willReturn(TAX_FACTOR); // when double meanTaxFactor = systemUnderTest.calculateMeanTaxFactorFor(new Person()); // then then(meanTaxFactor).isEqualTo(TAX_FACTOR); } } To profit from Mockito's annotations using JUnit, you just have to annotate your test class with @RunWith(MockitoJUnitRunner.class). How it works... The Mockito test runner will adapt its strategy depending on the version of JUnit. If there exists a org.junit.runners.BlockJUnit4ClassRunner class, it means that the codebase is using at least JUnit in Version 4.5.What eventually happens is that the MockitoAnnotations.initMocks(...) method is executed for the given test, which initializes all the Mockito annotations (for more information, check the subsequent There's more… section). There's more... You may have a situation where your test class has already been annotated with a @RunWith annotation and seemingly, you may not profit from Mockito's annotations. In order to achieve this, you have to call the MockitoAnnotations.initMocks method manually in the @Before annotated method of your test, as shown in the following code: public class MeanTaxFactorCalculatorTest { static final double TAX_FACTOR = 10; @Mock TaxService taxService; @InjectMocks MeanTaxFactorCalculator systemUnderTest; @Before public void setup() { MockitoAnnotations.initMocks(this); } @Test public void should_calculate_mean_tax_factor() { // given given(taxService.getCurrentTaxFactorFor(Mockito.any(Person.class))).willReturn(TAX_FACTOR); // when double meanTaxFactor = systemUnderTest.calculateMeanTaxFactorFor(new Person()); // then then(meanTaxFactor).isEqualTo(TAX_FACTOR); } } To use Mockito's annotations without a JUnit test runner, you have to call the MockitoAnnotations.initMocks method and pass the test class as its parameter. Mockito checks whether the user has overridden the global configuration of AnnotationEngine, and if this is not the case, the InjectingAnnotationEngine implementation is used to process annotations in tests. What is done internally is that the test class fields are scanned for annotations and proper test doubles are initialized and injected into the @InjectMocks annotated object (either by a constructor, property setter, or field injection, in that precise order). You have to remember several factors related to the automatic injection of test doubles as follows: If Mockito is not able to inject test doubles into the @InjectMocks annotated fields through either of the strategies, it won't report failure—the test will continue as if nothing happened (and most likely, you will get NullPointerException). For constructor injection, if arguments cannot be found, then null is passed For constructor injection, if nonmockable types are required in the constructor, then the constructor injection won't take place. For other injection strategies, if you have properties with the same type (or same erasure) and if Mockito matches mock names with a field/property name, it will inject that mock properly. Otherwise, the injection won't take place. For other injection strategies, if the @InjectMocks annotated object wasn't previously initialized, then Mockito will instantiate the aforementioned object using a no-arg constructor if applicable. See also JUnit documentation at https://github.com/junit-team/junit/wiki Martin Fowler's article on xUnit at http://www.martinfowler.com/bliki/Xunit.html Gerard Meszaros's xUnit Test Patterns at http://xunitpatterns.com/ @InjectMocks Mockito documentation (with description of injection strategies) at http://docs.mockito.googlecode.com/hg/1.9.5/org/mockito/InjectMocks.html Getting started with Mockito for TestNG Before going into details regarding Mockito and TestNG integration, it is worth mentioning a few words about TestNG. TestNG is a unit testing framework for Java that was created, as the author defines it on the tool's website (refer to the See also section for the link), out of frustration for some JUnit deficiencies. TestNG was inspired by both JUnit and TestNG and aims at covering the whole scope of testing—from unit, through functional, integration, end-to-end tests, and so on. However, the JUnit library was initially created for unit testing only. The main differences between JUnit and TestNG are as follows: The TestNG author disliked JUnit's approach of having to define some methods as static to be executed before the test class logic gets executed (for example, the @BeforeClass annotated methods)—that's why in TestNG you don't have to define these methods as static TestNG has more annotations related to method execution before single tests, suites, and test groups TestNG annotations are more descriptive in terms of what they do; for example, the JUnit's @Before versus TestNG's @BeforeMethod Mockito in Version 1.9.5 doesn't provide any out-of-the-box solution to integrate with TestNG in a simple way, but there is a special Mockito subproject for TestNG (refer to the See also section for the URL) that should be part one of the subsequent Mockito releases. In the following recipe, we will take a look at how to profit from that code and that very elegant solution. Getting ready When you take a look at Mockito's TestNG subproject on the Mockito GitHub repository, you will find that there are three classes in the org.mockito.testng package, as follows: MockitoAfterTestNGMethod MockitoBeforeTestNGMethod MockitoTestNGListener Unfortunately, until this project eventually gets released you have to just copy and paste those classes to your codebase. How to do it... To integrate TestNG and Mockito, perform the following steps: Copy the MockitoAfterTestNGMethod, MockitoBeforeTestNGMethod, and MockitoTestNGListener classes to your codebase from Mockito's TestNG subproject. Annotate your test class with @Listeners(MockitoTestNGListener.class). Annotate the test fields with the @Mock or @Spy annotation to have either a mock or spy object instantiated. Annotate the test fields with the @InjectMocks annotation to first instantiate the @InjectMock annotated field and inject all the @Mock or @Spy annotated fields into it (if applicable). Annotate the test fields with the @Captor annotation to make Mockito instantiate an argument captor. Now let's take a look at this snippet that, using TestNG, checks whether the mean tax factor value has been calculated properly (remember that I'm using the BDDMockito.given(...) and AssertJ's BDDAssertions.then(...) static methods: @Listeners(MockitoTestNGListener.class) public class MeanTaxFactorCalculatorTestNgTest { static final double TAX_FACTOR = 10; @Mock TaxService taxService; @InjectMocks MeanTaxFactorCalculator systemUnderTest; @Test public void should_calculate_mean_tax_factor() { // given given(taxService.getCurrentTaxFactorFor(any(Person.class))).willReturn(TAX_FACTOR); // when double meanTaxFactor = systemUnderTest.calculateMeanTaxFactorFor(new Person()); // then then(meanTaxFactor).isEqualTo(TAX_FACTOR); } } How it works... TestNG allows you to register custom listeners (your listener class has to implement the IInvokedMethodListener interface). Once you do this, the logic inside the implemented methods will be executed before and after every configuration and test methods get called. Mockito provides you with a listener whose responsibilities are as follows: Initialize mocks annotated with the @Mock annotation (it is done only once) Validate the usage of Mockito after each test method Remember that with TestNG, all mocks are reset (or initialized if it hasn't already been done so) before any TestNG method! See also The TestNG homepage at http://testng.org/doc/index.html The Mockito TestNG subproject at https://github.com/mockito/mockito/tree/master/subprojects/testng The Getting started with Mockito for JUnit recipe on the @InjectMocks analysis

Background music and audio effects play a big role in determining any game's success or failure. Creating engaging game audio, importing audio from other sources and working and customizing Audio FX clips as per the game flow is a vital task for any game developer.  In this article, we are going to discuss about how to create, customize and use third party audio in Unity games. This article is a part of the book titled "Unity 2017 2D Game Development Projects" written by Lauren S. Ferro & Francesco Sapio. Basics of audio and sound FX in Unity Adding sound in Unity is simple enough, but you can implement it better if you understand how sound travels. While this is extremely important in 3D games because of the added third dimension, it is quite important in 2D games, just in a slightly different way. Before we discuss the differences, let's first learn about what and how sound works from a quick physics lesson. Listening to the physics behind sound What we hear is not just music, sound effects (FX) and ambient background noise. The sound is a longitudinal, mechanical (vibrating) wave. These "waves" can pass through different mediums (for example, air, water, your desk) but not through a vacuum. Therefore, no one will hear your screams in space. The sound is a variation in pressure. A region of increased pressure on a sound wave is called a compression (or condensation). A region of decreased pressure on a sound wave is called a rarefaction (or dilation). You can see this concept illustrated in the following image: The density of certain materials, such as glass and plastic, allows a certain amount of light to pass through them. This will influence how the light will behave when it passes through them, such as bending/refracting (that is, the index of refraction), various materials (for example, liquids, solids, gases) have the same effect when it comes to allowing sound waves to pass. Some materials allow the sound to pass easily, while others dampen it. Therefore, sound studios/booths are made of certain materials to remove things such as echoes. It has a similar effect to when you scream underwater that there is a shark. It won't be as loud as if you scream from your kitchen to tell everyone dinner is ready. Another thing to consider is what is known as the Doppler Effect. The Doppler Effect results from an increase (or decrease) in the frequency of sound (and other things such as light, ripples in water) as the source of the sound and person/player move toward (or away from) each other. A simple example of this is when an emergency vehicle passes by you. You will notice that the sound of the siren is different before it reaches you when it is near you, and once it passes you. Considering this example, it is because there is a sudden change in pitch in the passing siren. This is visualized in the following image: So, what is the point of knowing this when it comes to developing games? Well, this is particularly important when creating games, more so in 3D, in relation to how sounds are heard by players in many ways. For example, imagine that you're nearing a creek, but there are dense bushes, large pine trees, and a rugged terrain. The sound that creek makes from where a player is in the game world is going to sound very different if it was a completely flat plane free from any vegetation. When it comes to 2D games, this is not necessarily as important because we are working without depth (z-axis) but similar principles apply when players may be navigating around a top-down environment and they are near a point of interest. You don't want that sound to be as loud when the player is far away as it would be if they were up close. Within the context of 2D and 3D sounds, Unity has a parameter for this exact thing called Spatial Blend. We will discuss this more in the Audio Source section. There are several ways that you can create audio within Unity, from importing your own/downloaded sounds to recording it live. Like images, Unity can import most standard audio file formats: AIFF, WAV, MP3, and Ogg, and tracker modules (for example, short instrument samples): .xm, .mod, .it, and .s3m. Importing audio Importing audio into Unity follows the same processes as importing any other type of asset. We will cover the basics of what you need to know in the following sections. Audio Listener Have you heard the saying, If a tree falls in a forest and no one is there to hear it, does it still make a sound? Well, in Unity, if there is nothing to hear your audio, then the answer is no. This is because Unity has a component called an Audio Listener, which works like a microphone. To locate the Audio Listener, click the Main Camera, and then look over at the Inspector; it should be located near the bottom, like in the following image: If for some reason, it isn't there, you can always add it by clicking the following button titled Add Component, type Audio Listener, and select it (click it) from the list, like in the following image: The important thing to remember is that an Audio Listener is the location of the sound, so it makes sense as to why it is typically placed on the Main Camera, but it can also be placed on a Player. A single scene can only have one Audio Listener; therefore, it's best to experiment with the one that works best for your game. It is important to remember that an Audio Listener works with an Audio Source, and must have one to work. Audio Source The Audio Source is where the sound comes from. This can be from many different objects within a Scene as well as background and sound FX. The Audio Source has several parameters; later we will briefly discuss the main ones. To see more information about all the parameters, you can check out the official Unity documentation by visiting the link or scanning the QR code: https://docs.unity3d.com/2017.2/Documentation/Manual/class-AudioSource.html You may be wondering why we should have a slider for Spatial Blend, instead of a checkbox. This is because we need to fade between 2D and 3D, and there is a good reason for this. Imagine that you're in a game and you're looking at a screen on a computer. In this case, your camera is going to be fixated on whatever is on the screen. This could be checking an inventory or even entering nuclear codes. In any case, you will want the sound that is being emitted from the screen to be the focal audio. Therefore, the slider in the Spatial Blend parameter is going to be closer to 2D. This is because you may still want ambient noises that are in the background incorporated into the experience. So, if you are closer to 2D, the sound will be the same in both speakers (or headphones). The closer you slide toward 3D, the more the volume will depend on the proximity of the Sound Listener to the Sound Source. It will also allow for things, such as the Doppler Effect, to be more noticeable, as it takes in 3D space. There are also specific settings for these things. Choosing sounds for background and FX When it comes to picking the right kind of music for your game, just like the aesthetics, you need to think about what kind of "mood" you're trying to create. Is it a somber or uplifting kind of mood, are you ironically contrasting the graphics (for example, happy) with gloomy music? There is really no right or wrong when it comes to your musical selection if you can communicate to the player what they are supposed to feel, at least in general. For this game, I have provided you with some example "moods" that you can apply to this game. Of course, you're welcome to choose sounds other than this that are more to your liking! All the sounds that we will use will be from the Free Sound website: https://freesound.org. You will need to create an account to download them, but it's free and there are many great sounds that you can use when creating games. In saying this, if you're intending to create your games for commercial purposes, please make sure that you check the Terms and Conditions on Free Sound to make sure that you're not violating any of them. Each track will have its own attribution licenses, including those for commercial use, so always check! For this project, we're going to stick with the "Happy" version. But I encourage you to experiment! Happy Collecting Angel Cakes: Chime sound (https://freesound.org/people/jgreer/sounds/333629/) Being attacked by the enemy: Cat Purr/Twit4.wav (https://freesound.org/people/steffcaffrey/sounds/262309/) Collecting health: correct (https://freesound.org/people/ertfelda/sounds/243701/) Collecting bonuses: Signal-Ring 1 (https://freesound.org/people/Vendarro/sounds/399315/) Background: Kirmes_Orgel_004_2_Rosamunde.mp3 (https://freesound.org/people/bilwiss/sounds/24720/) Sad Collecting Angel Cakes: Glass Tap (https://freesound.org/people/Unicornaphobist/sounds/262958/) Being attacked by the enemy: musicbox1.wav (https://freesound.org/people/sandocho/sounds/17700/) Collecting health: chime.wav (https://freesound.org/people/Psykoosiossi/sounds/398661/) Collecting bonuses: short metallic hit (https://freesound.org/people/waveplay/sounds/366400/) Background: improvised chill 8 (https://freesound.org/people/waveplay/sounds/238529/) Retro Collecting Angel Cakes: TF_Buzz.flac (https://freesound.org/people/copyc4t/sounds/235652/) Being attacked by the enemy: Game Die (https://freesound.org/people/josepharaoh99/sounds/364929/) Collecting health: galanghee.wav (https://freesound.org/people/metamorphmuses/sounds/91387/) Collecting bonuses: SW05.WAV (https://freesound.org/people/mad-monkey/sounds/66684/) Background: Angel-techno pop music loop (https://freesound.org/people/frankum/sounds/387410/) Not everyone can hear well or at all, so it pays to keep this in mind when you're developing games that may rely on audio to provide feedback to players. While subtitles can enable dialogue to be more accessible, sound FX can be a little trickier. Therefore, when it comes to implementing audio, think about how you could complement it, even if the same effect that you're trying to achieve with sound is subtle. For example, if you play a "bleep" for every item collected, perhaps you could associate it with a slight glow or flash of color. The choice is up to you, but it's something to keep in mind. On the other end of the spectrum, those who can hear might also want to turn the sounds off. We've all played that game (or several) that really begins to become irritating, so make sure that you also check this while you're playtesting. You don't want an awesome game to suck because your audio is intolerable and there is not an option to TURN THE SOUND OFF! You’ve been warned. Integrating background music in our game Once you choose which music better suits the kind of feel you want to create for your game, import both the sound and the music inside the project. If you want, you can create two folders for them, SoundFX and Music, respectively. Now, in our scene, we need to do the following: Create an empty game object (by clicking GameObject | Create empty), rename it Background Music. Attach an Audio Source component (in the Inspector, click Add Component | Audio | Audio Source). Next, we need to drag and drop the music we decided on/downloaded into the AudioClip variable and check the Loop option, so the background music will never stop. Also, check that Play on Awake is checked as well, even if it should be by default, so the music will start playing as soon as the game starts. Hit Play to start the game. Lastly, adjust the volume, depending on the music you chose. This may require a bit of playtesting (remember to set the value after the play mode, because the settings you adjust during play mode are not kept). In the end, this is how the component should look (in the image, I chose the happy theme music, and set a Volume of 0.1): Here in this article we have shown you how to incorporate game audio effects and background music in Unity games. If you liked this article, then check out the complete book Unity 2017 2D Game Development Projects. AI for Unity game developers: How to emulate real-world senses in your NPC agent Working with Unity Variables to script powerful Unity 2017 games How to use arrays, lists, and dictionaries in Unity for 3D game development

Google researchers have unveiled a new real-time hand tracking algorithm that could be a new breakthrough for people communicating via sign language. Their algorithm uses machine learning to compute 3D keypoints of a hand from a video frame. This research is implemented in MediaPipe which is an open-source cross-platform framework for building multimodal (eg. video, audio, any time series data) applied ML pipelines. What is interesting is that the 3D hand perception can be viewed in real-time on a mobile phone. How real-time hand perception and gesture recognition works with MediaPipe? The algorithm is built using the MediaPipe framework. Within this framework, the pipeline is built as a directed graph of modular components. The pipeline employs three different models: a palm detector model, a handmark detector model and a gesture recognizer. The palm detector operates on full images and outputs an oriented bounding box. They employ a single-shot detector model called BlazePalm, They achieve an average precision of 95.7% in palm detection. Next, the hand landmark takes the cropped image defined by the palm detector and returns 3D hand keypoints. For detecting key points on the palm images, researchers manually annotated around 30K real-world images with 21 coordinates. They also generated a synthetic dataset to improve the robustness of the hand landmark detection model. The gesture recognizer then classifies the previously computed keypoint configuration into a discrete set of gestures. The algorithm determines the state of each finger, e.g. bent or straight, by the accumulated angles of joints. The existing pipeline supports counting gestures from multiple cultures, e.g. American, European, and Chinese, and various hand signs including “Thumb up”, closed fist, “OK”, “Rock”, and “Spiderman”. They also trained their models to work in a wide variety of lighting situations and with a diverse range of skin tones. Gesture recognition - Source: Google blog With MediaPipe, the researchers built their pipeline as a directed graph of modular components, called Calculators. Individual calculators like cropping, rendering , and neural network computations can be performed exclusively on the GPU. They employed TFLite GPU inference on most modern phones. The researchers are open sourcing the hand tracking and gesture recognition pipeline in the MediaPipe framework along with the source code. The researchers Valentin Bazarevsky and Fan Zhang write in a blog post, “Whereas current state-of-the-art approaches rely primarily on powerful desktop environments for inference, our method, achieves real-time performance on a mobile phone, and even scales to multiple hands. We hope that providing this hand perception functionality to the wider research and development community will result in an emergence of creative use cases, stimulating new applications and new research avenues.” People commended the fact that this algorithm can run on mobile devices and is useful for people who communicate via sign language. https://twitter.com/SOdaibo/status/1163577788764495872 https://twitter.com/anshelsag/status/1163597036442148866 https://twitter.com/JonCorey1/status/1163997895835693056 Microsoft Azure VP demonstrates Holoportation, a reconstructed transmittable 3D technology Terrifyingly realistic Deepfake video of Bill Hader transforming into Tom Cruise is going viral on YouTube. Google News Initiative partners with Google AI to help ‘deep fake’ audio detection research

Today, you will learn how to collect Bitcoin historical and live-price data. You will also learn to transform data into time series and train your model to make insightful predictions. Historical and live-price data collection We will be using the Bitcoin historical price data from Kaggle. For the real-time data, Cryptocompare API will be used. Historical data collection For training the ML algorithm, there is a Bitcoin  Historical  Price Data dataset available to the public on Kaggle (version 10). The dataset can be downloaded here. It has 1 minute OHLC data for BTC-USD pairs from several exchanges. At the beginning of the project, for most of them, data was available from January 1, 2012 to May 31, 2017; but for the Bitstamp exchange, it's available until October 20, 2017 (as well as for Coinbase, but that dataset became available later): Figure 1: The Bitcoin  historical dataset on Kaggle Note that you need to be a registered user and be logged in in order to download the file. The file that we are using is bitstampUSD_1-min_data_2012-01-01_to_2017-10-20.csv. Now, let us get the data we have. It has eight columns: Timestamp: The time elapsed in seconds since January 1, 1970. It is 1,325,317,920 for the first row and 1,325,317,920 for the second 1. (Sanity check! The difference is 60 seconds). Open: The price at the opening of the time interval. It is 4.39 dollars. Therefore it is the price of the first trade that happened after Timestamp (1,325,317,920 in the first row's case). Close: The price at the closing of the time interval. High: The highest price from all orders executed during the interval. Low: The same as High but it is the lowest price. Volume_(BTC): The sum of all Bitcoins that were transferred during the time interval. So, take all transactions that happened during the selected interval and sum up the BTC values of each of them. Volume_(Currency): The sum of all dollars transferred. Weighted_Price: This is derived from the volumes of BTC and USD. By dividing all dollars traded by all bitcoins, we can get the weighted average price of BTC during this minute. So Weighted_Price=Volume_(Currency)/Volume_(BTC). One of the most important parts of the data-science pipeline after data collection (which is in a sense outsourced; we use data collected by others) is data preprocessing—clearing a dataset and transforming it to suit our needs. Transformation of historical data into a time series Stemming from our goal—predict the direction of price change—we might ask ourselves, does having an actual price in dollars help to achieve this? Historically, the price of Bitcoin was usually rising, so if we try to fit a linear regression, it will show further exponential growth (whether in the long run this will be true is yet to be seen). Assumptions and design choices One of the assumptions of this project is as follows: whether we are thinking about Bitcoin trading in November 2016 with a price of about $700, or trading in November 2017 with a price in the $6500-7000 range, patterns in how people trade are similar. Now, we have several other assumptions, as described in the following points: Assumption one: From what has been said previously, we can ignore the actual price and rather look at its change. As a measure of this, we can take the delta between opening and closing prices. If it is positive, it means the price grew during that minute; the price went down if it is negative and stayed the same if delta = 0. In the following figure, we can see that Delta was -1.25 for the first minute observed, -12.83 for the second one, and -0.23 for the third one. Sometimes, the open price can differ significantly from the close price of the previous minute (although Delta is negative during all three of the observed minutes, for the third minute the shown price was actually higher than close for a second). But such things are not very common, and usually the open price doesn't change significantly compared to the close price of the previous minute. Assumption two: The next need to consider...  is predicting the price change in a black box environment. We do not use other sources of knowledge such as news, Twitter feeds, and others to predict how the market would react to them. This is a more advanced topic. The only data we use is price and volume. For simplicity of the prototype, we can focus on price only and construct time series data. Time series prediction is a prediction of a parameter based on the values of this parameter in the past. One of the most common examples is temperature prediction. Although there are many supercomputers using satellite and sensor data to predict the weather, a simple time series analysis can lead to some valuable results. We predict the price at T+60 seconds, for instance, based on the price at T, T-60s, T-120s and so on. Assumption three: Not all data in the dataset is valuable. The first 600,000 records are not informative, as price changes are rare and trading volumes are small. This can affect the model we are training and thus make end results worse. That is why the first 600,000 of rows are eliminated from the dataset. Assumption four: We need to Label  our data so that we can use a supervised ML algorithm. This is the easiest measure, without concerns about transaction fees. Data preprocessing Taking into account the goals of data preparation, Scala was chosen as an easy and interactive way to manipulate data: val priceDataFileName: String = "bitstampUSD_1- min_data_2012-01-01_to_2017-10-20.csv" val spark = SparkSession .builder() .master("local[*]") .config("spark.sql.warehouse.dir", "E:/Exp/") .appName("Bitcoin Preprocessing") .getOrCreate() val data = spark.read.format("com.databricks.spark.csv").option("header", "true").load(priceDataFileName) data.show(10) >>> println((data.count(),  data.columns.size)) >>> (3045857,  8) In the preceding code, we load data from the file downloaded from Kaggle and look at what is inside. There are 3045857 rows in the dataset and 8 columns, described before. Then we create the Delta column, containing the difference between closing and opening prices (that is, to consider only that data where meaningful trading has started to occur): val dataWithDelta  = data.withColumn("Delta",  data("Close") - data("Open")) The following code labels our data by assigning 1 to the rows the Delta value of which was positive; it assigns 0 otherwise: import org.apache.spark.sql.functions._  import spark.sqlContext.implicits._  val dataWithLabels  = dataWithDelta.withColumn("label",    when($"Close"  -  $"Open"  > 0, 1).otherwise(0))  rollingWindow(dataWithLabels,  22, outputDataFilePath,  outputLabelFilePath) This code transforms the original dataset into time series data. It takes the Delta values of WINDOW_SIZE rows (22 in this experiment) and makes a new row out of them. In this way, the first row has Delta values from t0 to t21, and the second one has values from t1 to t22. Then we create the corresponding array with labels (1 or 0). Finally, we save X and Y into files where 612000 rows were cut off from the original dataset; 22 means rolling window size and 2 classes represents that labels are binary 0 and 1: val dropFirstCount: Int = 612000 def rollingWindow(data: DataFrame, window: Int, xFilename: String, yFilename: String): Unit = { var i = 0 val xWriter = new BufferedWriter(new FileWriter(new File(xFilename))) val yWriter = new BufferedWriter(new FileWriter(new File(yFilename))) val zippedData = data.rdd.zipWithIndex().collect() System.gc() val dataStratified = zippedData.drop(dropFirstCount)//slice 612K while (i < (dataStratified.length - window)) { val x = dataStratified .slice(i, i + window) .map(r => r._1.getAs[Double]("Delta")).toList val y = dataStratified.apply(i + window)._1.getAs[Integer]("label") val stringToWrite = x.mkString(",") xWriter.write(stringToWrite + "n") yWriter.write(y + "n") i += 1 if (i % 10 == 0) { xWriter.flush() yWriter.flush() } } xWriter.close() yWriter.close() } In the preceding code segment: val outputDataFilePath:  String = "output/scala_test_x.csv" val outputLabelFilePath:  String = "output/scala_test_y.csv" Real-time data through the Cryptocompare API For real-time data, the Cryptocompare API is used, more specifically HistoMinute, which gives us access to OHLC data for the past seven days at most. The details of the API will be discussed in a section devoted to implementation, but the API response is very similar to our historical dataset, and this data is retrieved using a regular HTTP request. For example, a simple JSON response has the following structure: {  "Response":"Success", "Type":100, "Aggregated":false, "Data":  [{"time":1510774800,"close":7205,"high":7205,"low":7192.67,"open":7198,  "volumefrom":81.73,"volumeto":588726.94},  {"time":1510774860,"close":7209.05,"high":7219.91,"low":7205,"open":7205, "volumefrom":16.39,"volumeto":118136.61},    ... (other  price data)    ], "TimeTo":1510776180,    "TimeFrom":1510774800,    "FirstValueInArray":true,    "ConversionType":{"type":"force_direct","conversionSymbol":""} } Through Cryptocompare HistoMinute, we can get open, high, low, close, volumefrom, and volumeto from each minute of historical data. This data is stored for 7 days only; if you need more, use the hourly or daily path. It uses BTC conversion if data is not available because the coin is not being traded in the specified currency: Now, the following method fetches the correctly formed URL of the Cryptocompare API, which is a fully formed URL with all parameters, such as currency, limit, and aggregation specified. It finally returns the future that will have a response body parsed into the data model, with the price list to be processed at an upper level: import javax.inject.Inject import play.api.libs.json.{JsResult, Json} import scala.concurrent.Future import play.api.mvc._ import play.api.libs.ws._ import processing.model.CryptoCompareResponse class RestClient @Inject() (ws: WSClient) { def getPayload(url : String): Future[JsResult[CryptoCompareResponse]] = { val request: WSRequest = ws.url(url) val future = request.get() implicit val context = play.api.libs.concurrent.Execution.Implicits.defaultContext future.map { response => response.json.validate[CryptoCompareResponse] } } } In the preceding code segment, the CryptoCompareResponse class is the model of API, which takes the following parameters: Response Type Aggregated\ Data FirstValueInArray TimeTo TimeFrom Now, it has the following signature: case class CryptoCompareResponse(Response  : String, Type : Int,  Aggregated  : Boolean, Data  :  List[OHLC], FirstValueInArray  :  Boolean, TimeTo  : Long,  TimeFrom:  Long) object CryptoCompareResponse  {  implicit  val cryptoCompareResponseReads  = Json.reads[CryptoCompareResponse] } Again, the preceding two code segments the open-high-low-close (also known as OHLC), are a model class for mapping with CryptoAPI response data array internals. It takes these parameters: Time: Timestamp in seconds, 1508818680, for instance. Open: Open price at a given minute interval. High: Highest price. Low: Lowest price. Close: Price at the closing of the interval. Volumefrom: Trading volume in the from currency. It's BTC in our case. Volumeto: The trading volume in the to currency, USD in our case. Dividing Volumeto by Volumefrom gives us the weighted price of BTC. Now, it has the following signature: case class OHLC(time:  Long,  open:  Double,  high:  Double,  low:  Double,  close:  Double,  volumefrom:  Double,  volumeto:  Double)  object OHLC  {    implicit  val implicitOHLCReads  = Json.reads[OHLC]  } Model training for prediction Inside the project, in the package folder prediction.training, there is a Scala object called TrainGBT.scala. Before launching, you have to specify/change four things: In the code, you need to set up spark.sql.warehouse.dir in some actual place on your computer that has several gigabytes of free space: set("spark.sql.warehouse.dir",  "/home/user/spark") The RootDir is the main folder, where all files and train models will be stored:rootDir  = "/home/user/projects/btc-prediction/" Make sure that the x filename matches the one produced by the Scala script in the preceding step: x  = spark.read.format("com.databricks.spark.csv").schema(xSchema).load(rootDir  + "scala_test_x.csv") Make sure that the y filename matches the one produced by Scala script: y_tmp=spark.read.format("com.databricks.spark.csv").schema (ySchema).load(rootDir + "scala_test_y.csv") The code for training uses the Apache Spark ML library (and libraries required for it) to train the classifier, which means they have to be present in your class path to be able to run it. The easiest way to do that (since the whole project uses SBT) is to run it from the project root folder by typing sbtrun-main  prediction.training.TrainGBT, which will resolve all dependencies and launch training. Depending on the number of iterations and depth, it can take several hours to train the model. Now let us see how training is performed on the example of the gradient-boosted trees model. First, we need to create a SparkSession object: val xSchema = StructType(Array( StructField("t0", DoubleType, true), StructField("t1", DoubleType, true), StructField("t2", DoubleType, true), StructField("t3", DoubleType, true), StructField("t4", DoubleType, true), StructField("t5", DoubleType, true), StructField("t6", DoubleType, true), StructField("t7", DoubleType, true), StructField("t8", DoubleType, true), StructField("t9", DoubleType, true), StructField("t10", DoubleType, true), StructField("t11", DoubleType, true), StructField("t12", DoubleType, true), StructField("t13", DoubleType, true), StructField("t14", DoubleType, true), StructField("t15", DoubleType, true), StructField("t16", DoubleType, true), StructField("t17", DoubleType, true), StructField("t18", DoubleType, true), StructField("t19", DoubleType, true), StructField("t20", DoubleType, true), StructField("t21", DoubleType, true)) ) Then we read the files we defined for the schema. It was more convenient to generate two separate files in Scala for data and labels, so here we have to join them into a single DataFrame: Then we read the files we defined for the schema. It was more convenient to generate two separate files in Scala for data and labels, so here we have to join them into a single DataFrame: import spark.implicits._ val y = y_tmp.withColumn("y", 'y.cast(IntegerType)) import org.apache.spark.sql.functions._ val x_id = x.withColumn("id", monotonically_increasing_id()) val y_id = y.withColumn("id", monotonically_increasing_id()) val data = x_id.join(y_id, "id") The next step is required by Spark—we need to vectorize the features: val featureAssembler = new VectorAssembler() .setInputCols(Array("t0", "t1", "t2", "t3", "t4", "t5", "t6", "t7", "t8", "t9", "t10", "t11", "t12", "t13", "t14", "t15", "t16", "t17", "t18", "t19", "t20", "t21")) .setOutputCol("features") We split the data into train and test sets randomly in the proportion of 75% to 25%. We set the seed so that the splits would be equal among all times we run the training: val Array(trainingData,testData) = dataWithLabels.randomSplit(Array(0.75, 0.25), 123) We then define the model. It tells which columns are features and which are labels. It also sets parameters: val gbt = new GBTClassifier() .setLabelCol("label") .setFeaturesCol("features") .setMaxIter(10) .setSeed(123) Create a pipeline of steps—vector assembling of features and running GBT: val pipeline = new Pipeline() .setStages(Array(featureAssembler, gbt)) Defining evaluator function—how the model knows whether it is doing well or not. As we have only two classes that are imbalanced, accuracy is a bad measurement; area under the ROC curve is better: val rocEvaluator = new BinaryClassificationEvaluator() .setLabelCol("label") .setRawPredictionCol("rawPrediction") .setMetricName("areaUnderROC") K-fold cross-validation is used to avoid overfitting; it takes out one-fifth of the data at each iteration, trains the model on the rest, and then tests on this one-fifth: val cv = new CrossValidator() .setEstimator(pipeline) .setEvaluator(rocEvaluator) .setEstimatorParamMaps(paramGrid) .setNumFolds(numFolds) .setSeed(123) val cvModel = cv.fit(trainingData) After we get the trained model (which can take an hour or more depending on the number of iterations and parameters we want to iterate on, specified in paramGrid), we then compute the predictions on the test data: val predictions = cvModel.transform(testData) In addition, evaluate quality of predictions: val roc = rocEvaluator.evaluate(predictions) The trained model is saved for later usage by the prediction service: val gbtModel = cvModel.bestModel.asInstanceOf[PipelineModel] gbtModel.save(rootDir + "__cv__gbt_22_binary_classes_" + System.nanoTime()/ 1000000 + ".model") In summary, the code for model training is given as follows: import org.apache.spark.{ SparkConf, SparkContext } import org.apache.spark.ml.{ Pipeline, PipelineModel } import org.apache.spark.ml.classification.{ GBTClassificationModel, GBTClassifier, RandomForestClassificationModel, RandomForestClassifier} import org.apache.spark.ml.evaluation.{BinaryClassificationEvaluator, MulticlassClassificationEvaluator} import org.apache.spark.ml.feature.{IndexToString, StringIndexer, VectorAssembler, VectorIndexer} import org.apache.spark.ml.tuning.{CrossValidator, ParamGridBuilder} import org.apache.spark.sql.types.{DoubleType, IntegerType, StructField, StructType} import org.apache.spark.sql.SparkSession object TrainGradientBoostedTree { def main(args: Array[String]): Unit = { val maxBins = Seq(5, 7, 9) val numFolds = 10 val maxIter: Seq[Int] = Seq(10) val maxDepth: Seq[Int] = Seq(20) val rootDir = "output/" val spark = SparkSession .builder() .master("local[*]") .config("spark.sql.warehouse.dir", ""/home/user/spark/") .appName("Bitcoin Preprocessing") .getOrCreate() val xSchema = StructType(Array( StructField("t0", DoubleType, true), StructField("t1", DoubleType, true), StructField("t2", DoubleType, true), StructField("t3", DoubleType, true), StructField("t4", DoubleType, true), StructField("t5", DoubleType, true), StructField("t6", DoubleType, true), StructField("t7", DoubleType, true), StructField("t8", DoubleType, true), StructField("t9", DoubleType, true), StructField("t10", DoubleType, true), StructField("t11", DoubleType, true), StructField("t12", DoubleType, true), StructField("t13", DoubleType, true), StructField("t14", DoubleType, true), StructField("t15", DoubleType, true), StructField("t16", DoubleType, true), StructField("t17", DoubleType, true), StructField("t18", DoubleType, true), StructField("t19", DoubleType, true), StructField("t20", DoubleType, true), StructField("t21", DoubleType, true))) val ySchema = StructType(Array(StructField("y", DoubleType, true))) val x = spark.read.format("csv").schema(xSchema).load(rootDir + "scala_test_x.csv") val y_tmp = spark.read.format("csv").schema(ySchema).load(rootDir + "scala_test_y.csv") import spark.implicits._ val y = y_tmp.withColumn("y", 'y.cast(IntegerType)) import org.apache.spark.sql.functions._ //joining 2 separate datasets in single Spark dataframe val x_id = x.withColumn("id", monotonically_increasing_id()) val y_id = y.withColumn("id", monotonically_increasing_id()) val data = x_id.join(y_id, "id") val featureAssembler = new VectorAssembler() .setInputCols(Array("t0", "t1", "t2", "t3", "t4", "t5", "t6", "t7", "t8", "t9", "t10", "t11", "t12", "t13", "t14", "t15", "t16", "t17","t18", "t19", "t20", "t21")) .setOutputCol("features") val encodeLabel = udf[Double, String] { case "1" => 1.0 case "0" => 0.0 } val dataWithLabels = data.withColumn("label", encodeLabel(data("y"))) //123 is seed number to get same datasplit so we can tune params val Array(trainingData, testData) = dataWithLabels.randomSplit(Array(0.75, 0.25), 123) val gbt = new GBTClassifier() .setLabelCol("label") .setFeaturesCol("features") .setMaxIter(10) .setSeed(123) val pipeline = new Pipeline() .setStages(Array(featureAssembler, gbt)) // *********************************************************** println("Preparing K-fold Cross Validation and Grid Search") // *********************************************************** val paramGrid = new ParamGridBuilder() .addGrid(gbt.maxIter, maxIter) .addGrid(gbt.maxDepth, maxDepth) .addGrid(gbt.maxBins, maxBins) .build() val cv = new CrossValidator() .setEstimator(pipeline) .setEvaluator(new BinaryClassificationEvaluator()) .setEstimatorParamMaps(paramGrid) .setNumFolds(numFolds) .setSeed(123) // ************************************************************ println("Training model with GradientBoostedTrees algorithm") // ************************************************************ // Train model. This also runs the indexers. val cvModel = cv.fit(trainingData) cvModel.save(rootDir + "cvGBT_22_binary_classes_" + System.nanoTime() / 1000000 + ".model") println("Evaluating model on train and test data and calculating RMSE") // ********************************************************************** // Make a sample prediction val predictions = cvModel.transform(testData) // Select (prediction, true label) and compute test error. val rocEvaluator = new BinaryClassificationEvaluator() .setLabelCol("label") .setRawPredictionCol("rawPrediction") .setMetricName("areaUnderROC") val roc = rocEvaluator.evaluate(predictions) val prEvaluator = new BinaryClassificationEvaluator() .setLabelCol("label") .setRawPredictionCol("rawPrediction") .setMetricName("areaUnderPR") val pr = prEvaluator.evaluate(predictions) val gbtModel = cvModel.bestModel.asInstanceOf[PipelineModel] gbtModel.save(rootDir + "__cv__gbt_22_binary_classes_" + System.nanoTime()/1000000 +".model") println("Area under ROC curve = " + roc) println("Area under PR curve= " + pr) println(predictions.select().show(1)) spark.stop() } } Now let us see how the training went: >>> Area under ROC curve = 0.6045355104779828 Area under PR curve= 0.3823834607704922 Therefore, we have not received very high accuracy, as the ROC is only 60.50% out of the best GBT model. Nevertheless, if we tune the hyperparameters, we will get better accuracy. We learned how a complete ML pipeline can be implemented, from collecting historical data, to transforming it into a format suitable for testing hypotheses. We also performed machine learning model training to carry out predictions. You read an excerpt from a book written by Md. Rezaul Karim, titled Scala Machine Learning Projects. In this book, you will learn to build powerful machine learning applications for performing advanced numerical computing and functional programming.  

Creating virtual machines on Azure gives you on-demand, high-scale, secure, virtualized infrastructure using Windows Server. Virtual machine helps you deploy and scale applications easily. In this article, we will learn how to run an Azure Virtual Machine. This tutorial is an excerpt from the book, Hands-On Networking with Azure, written by Mohamed Waly. This book will help you efficiently monitor, diagnose, and troubleshoot Azure Networking. Creating an Azure VM is a very straightforward process – all you have to do is follow the given steps: Navigate to the Azure portal and search for Virtual Machines, as shown in the following screenshot: Figure 3.1: Searching for Virtual Machines Once the VM blade is opened, you can click on +Add to create a new VM, as shown in the following screenshot: Figure 3.2: Virtual Machines blade Once you have clicked on +Add, a new blade will pop up where you have to search for and select the desired OS for the VM, as shown in the following screenshot: Figure 3.3: Searching for Windows Server 2016 OS for the VM Once the OS is selected, you need to select the deployment model, whether that be Resource Manager or Classic, as shown in the following screenshot: Figure 3.4: Selecting the deployment model Once the deployment model is selected, a new blade will pop up where you have to specify the following: Name: Specify the name of the VM. VM disk type: Specify whether the disk type will be SSD or HDD. Consider that SSD will offer consistent, low-latency performance, but will incur more charges. Note that this option is not available for the Classic model in this blade, but is available in the Configure optional features blade. User name: Specify the username that will be used to log on the VM. Password: Specify the password, which must be between 12 and 123 characters long and must contain three of the following: one lowercase character, one uppercase character, one number, and one special character that is not or -. Subscription: This specifies the subscription that will be charged for the VM usage. Resource group: This specifies the resource group within which the VM will exist. Location: Specify the location in which the VM will be created. It is recommended that you select the nearest location to you. Save money: Here, you specify whether you own Windows Server Licenses with active Software Assurance (SA). If you do, Azure Hybrid Benefit is recommended to save compute costs. For more information about Azure Hybrid Benefit, you can check this page. Figure 3.5: Configure the VM basic settings Once you have clicked on OK, a new blade will pop up where you have to specify the VM size so that the VMs series can select the one that will fulfil your needs, as shown in the following screenshot: Figure 3.6: Select the VM size Once the VM size has been specified, you need to specify the following settings: Availability set: This option provides High availability for the VM by setting the VMs within the same application and availability set. Here, the VMs will be in different fault and update domains, granting the VMs high availability (up to 99.95% of Azure SLA). Use managed disks: Enable this feature to have Azure automatically manage the availability of disks to provide data redundancy and fault tolerance without creating and managing storage accounts on your own. This setting is not available in the Classic model. Virtual network: Specify the virtual network to which you want to assign the VM. Subnet: Select the subnet within the virtual network that you specified earlier to assign the VM to. Public IP address: Either select an existing public IP address or create a new one. Network security group (firewall): Select the NSG you want to assign to the VM NIC. This is called endpoints in the Classic model. Extensions: You can add more features to the VM using extensions, such as configuration management, antivirus protection, and so on. Auto-shutdown: Specify whether you want to shut down your VM daily or not; if you do, you can set a schedule. Considering that this option will help you saving compute cost especially for dev and test scenarios. This is not available in the Classic model. Notification before shutdown: Check this if you enabled Auto-shutdown and want to subscribe for notifications before the VM shuts down. This is not available in the Classic model. Boot diagnostics: This captures serial console output and screenshots of the VM running on a host to help diagnose start up issues. Guest OS diagnostics: This obtains metrics for the VM every minute; you can use these metrics to create alerts and stay informed of your applications. Diagnostics storage account: This is where metrics are written, so you can analyze them with your own tools. Figure 3.7: Specify more settings for the VM Enabling Boot diagnostics and Guest diagnostics will incur more charges since the diagnostics will need a dedicated storage account to store their data. Finally, once you are done with its settings, Azure will validate those you have specified and summarize them, as shown in the following screenshot: Figure 3.8: VM Settings Summary Once clicked on, Create the VM will start the creation process, and within minutes the VM will be created. Once the VM is created, you can navigate to the Virtual Machines blade to open the VM that has been created, as shown in the following screenshot: Figure 3.9: The created VM overview To connect to the VM, click on Connect, where a pre-configured RDP file with the required VM information will be downloaded. Open the RDP file. You will be asked to enter the username and password you specified for the VM during its configuration, as shown in the following screenshot: Figure 3.10: Entering the VM credentials Voila! You should now be connected to the VM. Azure VMs networking There are many network configurations that can be done for the VM. You can add additional NICs, change the private IP address, set a private or public IP address to be either static or dynamic, and you can change the inbound and outbound security rules. Adding inbound and outbound rules Adding inbound and outbound security rules to the VM NIC is a very simple process; all you need to do is follow these steps: Navigate to the desired VM. Scroll down to Networking, under SETTINGS, as shown in the following screenshot: Figure 3.11: VM networking settings To add inbound and outbound security rules, you have to click on either Add inbound or Add outbound. Once clicked on, a new blade will pop up where you have to specify settings using the following fields: Service: The service specifies the destination protocol and port range for this rule. Here, you can choose a predefined service, such as RDP or SSH, or provide a custom port range. Port ranges: Here, you need to specify a single port, a port range, or a comma-separated list of single ports or port ranges. Priority: Here, you enter the desired priority value. Name: Specify a name for the rule here. Description: Write a description for the rule that relates to it here. Figure 3.12: Adding an inbound rule Once you have clicked OK, the rule will be applied. Note that the same process applies when adding an outbound rule. Adding an additional NIC to the VM Adding an additional NIC starts from the same blade as adding inbound and outbound rules. To add an additional NIC, you have to follow the given steps: Before adding an additional NIC to the VM, you need to make sure that the VM is in a Stopped (Deallocated) status. Navigate to Networking on the desired VM. Click on Attach network interface, and a new blade will pop up. Here, you have to either create a network interface or select an existing one. If you are selecting an existing interface, simply click on OK and you are done. If you are creating a new interface, click on Create network interface, as shown in the following screenshot: Figure 3.13: Attaching network interface A new blade will pop up where you have to specify the following: Name: The name of the new NIC. Virtual network: This field will be grayed out because you cannot attach a VM's NIC to different virtual networks. Subnet: Select the desired subnet within the virtual network. Private IP address assignment: Specify whether you want to allocate this IP dynamically or statically. Network security group: Specify an NSG to be assigned to this NIC. Private IP address (IPv6): If you want to assign an IPv6 to this NIC, check this setting. Subscription: This field will be grayed out because you cannot have a VM's NIC in a different subscription. Resource group: Specify the resource group to which the NIC will exist. Location: This field will be grayed out because you cannot have VM NICs in different locations. Figure 3.14: Specify the NIC settings Once you are done, click Create. Once the network interface is created, you will return to the previous blade. Here, you need to specify the NIC you just created and click on OK, as shown in the following screenshot: Figure 3.15: Attaching the NIC Configuring the NICs The Network Interface Cards (NICs) include some configuration that you might be interested in. They are as follows: To navigate to the desired NIC, you can search for the network interfaces blade, as shown in the following screenshot: Figure 3.16: Searching for network interfaces blade Then, the blade will pop up, from which you can select the desired NIC, as shown in the following screenshot: Figure 3.17: Select the desired NIC You can also navigate back to the VM via | Networking | and then click on the desired NIC, as shown in the following screenshot: Figure 3.18: The VM NIC To configure the NIC, you need to follow the given steps: Once the NIC blade is opened, navigate to IP configurations, as shown in the following screenshot: Figure 3.19: NIC blade overview To enable IP forwarding, click on Enabled and then click Save. Enabling this feature will cause the NIC to receive traffic that is not destined to its own IP address. Traffic will be sent with a different source IP. To add another IP to the NIC, click on Add, and a new blade will pop up, for which you have to specify the following: Name: The name of the IP. Type: This field will be grayed out because a primary IP already exists. Therefore, this one will be secondary. Allocation: Specify whether the allocation method is static or dynamic. IP address: Enter the static IP address that belongs to the same subnet that the NIC belongs to. If you have selected dynamic allocation, you cannot enter the IP address statically. Public IP address: Specify whether or not you need a public IP address for this IP configuration. If you do, you will be asked to configure the required settings. Figure 3.20: Configure the IP configuration settings Click on Configure required settings for the public IP address and a new blade will pop up from which you can select an existing public IP address or create a new one, as shown in the following screenshot: Figure 3.21: Create a new public IP address Click on OK and you will return to the blade, as shown in Figure 3.20, with the following warning: Figure 3.22: Warning for adding a new IP address In this case, you need to plan for the addition of a new IP address to ensure that the time the VM is restarted is not during working hours. Azure VNets considerations for Azure VMs Building VMs in Azure is a common task, but to do this task well, and to make it operate properly, you need to understand the considerations of Azure VNets for Azure VMs. These considerations are as follows: Azure VNets enable you to bring your own IPv4/IPv6 addresses and assign them to Azure VMs, statically or dynamically You do not have access to the role that acts as DHCP or provides IP addresses; you can only control the ranges you want to use in the form of address ranges and subnets Installing a DHCP role on one of the Azure VMs is currently unsupported; this is because Azure does not use traditional Layer-2 or Layer-3 topology, and instead uses Layer-3 traffic with tunneling to emulate a Layer-2 LAN Private IP addresses can be used for internal communication; external communication can be done via public IP addresses You can assign multiple private and public IP addresses to a single VM You can assign multiple NICs to a single VM By default, all the VMs within the same virtual network can communicate with each other, unless otherwise specified by an NSG on a subnet within this virtual network The network security group (NSG) can sometimes cause an overhead; without this overhead, however, all VMs within the same subnet would communicate with each other By default, an inbound security rule is created for remote desktops for Windows-based VMs, and SSH for Linux-based VMs The inbound security rules are first applied on the NSG of the subnet and then the VM NIC NSG – for example, if the subnet's NSG allows HTTP traffic, it will pass through it; however, it may not reach its destination if the VM NIC NSG does not allow it The outbound security rules are applied for the VM NIC NSG first, and then applied on the subnet NSG Multiple NICs assigned to a VM can exist in different subnets Azure VMs with multiple NICs in the same availability set do not have to have the same number of NICs, but the VMs must have at least two NICs When you attach an NIC to a VM, you need to ensure that they exist in the same location and subscription The NIC and the VNet must exist in the same subscription and location The NIC's MAC address cannot be changed until the VM to which the NIC is assigned is deleted Once the VM is created, you cannot change the VNet to which it is assigned; however, you can change the subnet to which the VM is assigned You cannot attach an existing NIC to a VM during its creation, but you can add an existing NIC as an additional NIC By default, a dynamic public IP address is assigned to the VM during creation, but this address will change if the VM is stopped or deleted; to ensure it will not change, you need to ensure its IP address is static In a multi-NIC VM, the NSG that is applied to one NIC does not affect the others If you found this post useful, do check out the book Hands-On Networking with Azure, to design and implement Azure Networking for Azure VMs. Read More Introducing Azure Sphere – A secure way of running your Internet of Things devices Learn Azure Serverless computing for free: Download e-book

One of the most debated topics of the current mobile industry is the battle of the two giant app development platforms, Xamarin and React Native. Central to the buzzing hype of this battle and the increasing popularity of these two platforms are the communities of app developers built around them. Both of these open-source app development platforms are preferred by the app development community to create highly efficient applications while saving time and efforts of the app developers. Both React and Xamarin are supported by some merits and demerits, which makes selecting the best between the two a bit difficult. When it comes to selecting the appropriate mobile application platform, it boils down to the nature, needs and overall objectives of the business and company.  It also comes down to the features and characteristics of that technology, which either make it the best fit for one project or the worst approach for another. With that being said, let’s start with our comparing the two to find out the major differences, explore the key considerations and determine the winner of this unending platform battle. An overview of Xamarin An open-source cross-platform used for mobile application development, Xamarin can be used to build applications for Android, iOS and wearable devices. Offered as a high-tech enterprise app development tool within Microsoft Visual Studio IDE, Xamarin has now become one of the top mobile app development platforms used by various businesses and enterprises. Apart from being a free app development platform, it facilitates the development of mobile applications while using a single programming language, namely C#, for both the Android and iOS versions. Key features Since the day of its introduction, Xamarin has been using C#. C# is a popular programming language in the Microsoft community, and with great features like metaprogramming, functional programming and portability, C# is widely-preferred by many web developers. Xamarin makes it easy for C# developers to shift from web development platform to cross mobile app development platform. Features like portable class libraries, code sharing features, testing clouds and insights, and compatibility with Mac IDE and Visual Studio IDE makes Xamarin a great development tool with no additional costs. Development environment Xamarin provides app developers with a comprehensive app development toolkit and software package. The package includes highly compatible IDEs (for both Mac and VS), distribution and analytics tools such as Hockeyapp and testing tools such as Xamarin Test Cloud. With Xamarin, developers no longer have to invest their time and money in incorporating third-party tools. It uses Mono execution environment for both the platforms, i.e. Android and iOS. Framework C# has matured from its infancy, and the Xamarin framework now provides strong-safety typing which ensures prevention of unexpected code behavior. Since C# supports .NET framework, the language can be used with numerous .NET features like ASynC, LINQ, and Lambdas. Compilation Xamarin.iOS and Xamarin.Android are the two major products offered by this platform. In case of iOS code compilation, the platform follows Ahead-of-Time compilation whereas in Android Just-in-Time compilation approach is followed. However, the compilation process is fully automated and is equipped with features to tackle and resolve issues like memory allocation and garbage collection. App working principles Xamarin has an MVVM architecture coupled with a two-way data binding which provides great support for collaborative work among different departments. If your development approach doesn’t follow a strict performance-oriented approach, then go for Xamarin as it provides high process flexibility. How exactly does it work? Not only does C# form the basis of this platform, but it also provides developers with access to React Native APIs. This feature of Xamarin enables it to create universal backend code that can be used with any UI based on React Native SDK. An overview of React Native With Facebook being the creator of this platform, React Native is one of the widely-used programming platforms. From enabling mobile developers to build highly efficient apps to ensure great quality and increased sustainability, the demand for React Native apps is sure to increase over time. Key features React Native apps for the Android platform uses Java while the iOS version of the same app uses C#. The platforms provide numerous built-in tools, libraries, and frameworks. Its standout feature of hot reloading enables developers to make amendments to the code without spending much time on code compilation process. Development environment The React Native app development platform requires developers to follow a wide array of actions and processes to build a UI. The platform supports easy and faster iterations while enabling execution of a different code even when the application is running. Since React Native doesn’t provide support for 64-bit, it does impact the run time and speed of codes in iOS. Architecture React Native app development platform supports modular architecture. This means that developers can categorize the code into different functional and independent blocks of codes. This characteristic of the React Native platform, therefore, provides process flexibility, ease of upgrade and application updates. Compilation The Reactive Native app development platform follows and supports Just-in-Time compilation for Android applications. Whereas, in case of iOS application Just-in-Time compilation is not available as it might slow down the code execution procedure. App working principles This platform follows a one-way data binding approach which helps in boosting the overall performance of the application. However, through manual implementation, two-way data binding approach can be implemented which is useful for introducing code coherence and in reducing complex errors. How does it actually work? React Native enables developers to build applications using React and JavaScript. The working of a React Native application can be described as thread-based interaction. One thread handles the UI and user gestures while the other is React Native specific and deals with the application’s business logic. It also determines the structure and functionality of the overall user interface. The interaction could be asynchronous, batched or serializable. Learning curves of Xamarin and React Native To master Xamarin one has to be skilled in .NET. Xamarin provides you with easy and complete access to SDK platform capabilities because of Xamarin.iOS and Xamarin.Android libraries.  Xamarin provides a complete package which reduces the need of integrating third-party tools and libraries-- so to become a professional in Xamarin app development all you need is skills and expertise in C#, .NET and some basic working knowledge of React Native classes. While on the other hand, mastering React Native requires thorough knowledge and expertise of JavaScript. Since the platform doesn’t offer well-integrated libraries and tools, knowledge and expertise of third-party sources and tools are of core importance. Key differences between Xamarin and React Native While Trello, Slack, and GitHub use Xamarin, other successful companies like Facebook, Walmart, and Instagram have React Native-based mobile applications. While React, Native application offers better performance, not every company can afford to develop an app for each platform. Cross platforms like Xamarin are the best alternative to React Native apps as they offer higher development flexibility. Where Xamarin offers multiple platform support, cost-effectiveness and time-saving, React Native allows faster development and increased efficiency. Since Xamarin provides complete hardware support, the issues of hardware compatibility are reduced. React Native, on the other hand, provides you with ready-made components which reduce the need for writing the entire code from scratch. In React Native, with integration and after investment in third-party libraries and plugins, the need for WebView functions is eliminated which in turn reduces the memory requirements. Xamarin, on the other hand, provides you with a comprehensive toolkit with zero investments on additional plugins and third-party sources. However, this cross-platform offers restricted access to open-source technologies. A good quality React Native application requires more than a few weeks to develop which increases not only the development time but also the app complexity. If time-consumption is one of the drawbacks of the React Native app, then additional optimization for supporting larger application counts as a limitation for Xamarin. While frequent update contributes in shrinkage of the customer base of the React Native app, then stability complaints and app crashes are some common issues with Xamarin applications. When to go for Xamarin? Case #1: The foremost advantage of Xamarin is that all you need is command over C# and .NET. Case #2: One of the most exciting trends currently in the mobile development industry is the Internet of Things. Considering the rapid increase in need and demand of IoT, if you are developing a product that involves multiple hardware capacities and user devices then make developing with Xamarin your number one priority. Xamarin is fully compatible with numerous IoT devices which eliminates the need for a third-party source for functionality implementation. Case #3: If you are budget-constricted and time-bound then Xamarin is the solution to all your app development worries. Since the backend code for both Android and iOS is similar, it reduces the development time and efforts and is budget friendly. Case #4: The revolutionary and integral test cloud is probably the best part about Xamarin. Even though Test Cloud might take up a fraction of your budget, this expense is worth investing in. The test cloud not only recreates the activity of actual users but it also ensures that your application works well on various devices and is accessible to maximum users. When to go for React Native? Case #1: When it comes to game app development, Xamarin is not a wise choice. Since it supports C# framework and AOT compilation, getting speedy results and rendering is difficult with Xamarin. A Gaming application is updated dynamically, highly interactive and has high-performance graphics; the drawback of zero compatibility with heavy graphics makes Xamarin a poor choice in game app development. For these very reasons, many developers go for React Native when it comes to developing high-performing gaming applications. Case #2: The size of the application is an indirect indicator of the success of application among targeted users. Since many smartphone users have their own photos and video stuffed in their phone’s memory, there is barely any memory and storage left for an additional application. Xamarin-based apps are relatively heavier and occupy more space than their React Native counterparts. Wondering which framework to choose? Xamarin and React Native are the two major players of the mobile app development industry. So, it’s entirely up to you whether you want to proceed with React Native or Xamarin. However, your decision should be based on the type of application, requirements and development cost. If you want a faster development process go for the Xamarin and if you are developing a game, e-commerce or social site go for React Native. Author Bio Khalid Durrani is an Inbound Marketing Expert and a content strategist. He likes to cover the topics related to design, latest tech, startups, IOT, Artificial intelligence, Big Data, AR/VR, UI/UX and much more. Currently, he is the global marketing manager of LogoVerge, an AI-based design agency. The Ionic team announces the release of Ionic React Beta React Native 0.59 RC0 is now out with React Hooks, and more Changes made to React Native Community’s GitHub organization in 2018 for driving better collaboration

(For more resources related to this topic, see here.) Getting ready You can get the official ISO image file from https://www.archlinux.org/download/. On this page you will find a download link to the latest release. Depending on your preference, download the torrent file or the ISO image file immediately. The following list describes the main tasks that we will perform in this recipe: Preparing, booting, and setting keyboard layout: We are going to get the ISO file from the download page of the Arch Linux website and store it on the preferred media of our choice. At the time of writing this article, there is a dual ISO image file that contains both i686 and x86-64 architectures on one disk. Start your PC with your preferred installation media (CD or USB stick). On most PC systems, you can access the boot menu by pressing one of the function keys, usually between F8 and F12 depending on the motherboard manufacturer. On older machines where you do not yet have a boot menu, you might need to change the boot order in the BIOS where the CD-ROM (or DVD/Blu-ray) has to be chosen as the first device to try booting from. We'll also explain how to use a different keyboard layout than the default one in this recipe. Creating, formatting, and mounting partitions: You can partition the disks the way you want using cfdisk (for MBR disk partitioning) or cgdisk (for GUID disk partitioning). After creating the partitions, we can choose to format our created partitions with specific filesystems. When all partitions are formatted, we need to mount the partitions. First we will mount the root partition to /mnt. The other partitions will be mounted later on after you have created the specific folders. We'll designate our device with /dev/sdX; in your case this can be /dev/sda, and so on. Connecting to the Internet: To be able to continue installing the ISO you need to connect to the Internet, because there are no packages available for installation on the ISO. For a wireless network you will need to use netcfg. When connected to a wired network, just use dhcpcd or dhclient. Installing the base system and boot loader: These days the base system gets installed by running a simple script pacstrap. Pacstrap takes multiple parameters, the target location, and the packages or groups you want to install. For people who want to develop on their machines, the best base install is adding base-devel to the default installation. For normal end users, just base will be sufficient to start. Configuring the system: In this recipe, we'll describe the flow of what to do during the configuration. How to do it... The following steps will guide you in preparing, booting, and setting keyboard layout: Once you have downloaded the ISO image file, you should also verify its integrity by downloading the sha1sums.txt file from the download page. These days you can also check if the ISO is completely valid by verifying the signature of the ISO. Verify the integrity by issuing the sha1sum -c sha1sums.txt command and you'll see whether your download was successful or not. Also check if the signature of the ISO is correct by running gpg -v archlinux-...iso.sig: sha1sum -c sha1sums.txt gpg -v archlinux-2012-08-04-dual.iso.sig The following screenshot shows the execution of this step: As you can see in the previous screenshot, the ISO's checksum is ok and the signature is valid. Now that we are sure our ISO is ok, we can burn this to a CD with our favorite burning program. Insert the CD into the drive, or insert the USB stick into the USB port of your PC. Enter the boot menu, or let your computer automatically boot from the inserted installation media. If the previous steps are performed correctly, you will see the following screenshot: Select the architecture you want and press Enter, and we'll be on our way. Search the keyboard layout desired for your region. The available keyboard layouts can be found at /usr/share/kbd/keymaps/. Set the desired keyboard layout with loadkeys keyboardlayout. Now let's perform the following steps to create, format, and mount partitions: Start cfdisk or cgdisk, having the first parameter as the device you want to partition: cfdisk /dev/sdX cgdisk /dev/sdX Create your partition scheme. Store the partition scheme. Use the mkfs command to create a filesystem on a specific partition: mkfs -t vfat /dev/sdX mkfs.ext4 -L root /dev/sdX Mount your root partition to /mnt: mount /dev/sdX3 /mnt Make directories under mount for your other partitions: mkdir -p /mnt/boot Mount the other partitions: mount /dev/sdX1 /mnt/boot The following steps are needed to connect to the Internet: When we need a wireless network, create a netcfg profile and run netcfg mywireless. Use dhclient or dhcpcd to get an IP address. The following steps should be performed for installing the base system and boot loader: Run pacstrap with the desired parameters: pacstrap /mnt base base-devel Install the desired boot loader: the best choice at this moment is Syslinux. The final installation of the boot loader will be done in a chroot during the initial configuration. We'll now list the steps to do during the configuration: Generate fstab with genfstab: genfstab -p /mnt >> /mnt/etc/fstab Change the root into the system location: arch-chroot /mnt Set your hostname in /etc/hostname. Create /etc/localtime symlink. Set your locale in /etc/locale.conf. Uncomment the configured locale in /etc/locale.gen. Run locale-gen. Configure /etc/mkinitcpio.conf. Generate your initial ramdisk: mkinitcpio -p linux Finish installation of your boot loader. Set the root password with passwd. Leave the chroot environment (exit). How it works... We downloaded the ISO image file via torrent, or via HTTP from the mirror sites listed on the download page. The sha1sum command lets us verify the integrity of the downloaded ISO. On top of the checksum, we can also check the integrity by verifying the signature available for the ISO. So now, we can rest assured that the downloaded file is the real one. The ISO contains a fully working operating system. It also contains all the necessary tools to perform system recovery and installation. The keyboard configuration set with loadkeys will make sure that the key you press on your keyboard will be translated to the correct letter on your screen. Using a different keyboard layout from the one on your physical keyboard might be confusing. We then created a partition scheme on the selected disk with the appropriate tool (cfdisk or cgdisk). Make Filesystem (mkfs) is a unified frontend to create a filesystem. Using it we created our filesystem layout manually under/mnt by creating our default partition layout in our root, and mounting the specific partitions accordingly. You can make a connection with your wireless network (if needed), and then use dhcpcd or dhclient to obtain an IP address that enables you to access the Internet. Pacstrap will run pacman with a modified root location to install the desired packages into the newly created system. For example, installing Syslinux: pacstrap /mnt syslinux The specific configuration files will ensure we don't have to do all those steps over and over again on every boot. Summary This article explained the procedure to get Arch Linux installed on your system using the official installation media. Resources for Article : Further resources on this subject: Compression Formats in Linux Shell Script [Article] Making a Complete yet Small Linux Distribution [Article] Linux Shell Script: Tips and Tricks [Article]

In this article by Megan Squire, author of the book Clean Data, we will experiment with several data decanters to extract all the good stuff hidden inside inscrutable PDF files. We will explore the following topics: What PDF files are for and why it is difficult to extract data from them How to copy and paste from PDF files, and what to do when this does not work How to shrink a PDF file by saving only the pages that we need How to extract text and numbers from a PDF file using the tools inside a Python package called pdfMiner How to extract tabular data from within a PDF file using a browser-based Java application called Tabula How to use the full, paid version of Adobe Acrobat to extract a table of data (For more resources related to this topic, see here.) Why is cleaning PDF files difficult? Files saved in Portable Document Format (PDF) are a little more complicated than some of the text files. PDF is a binary format that was invented by Adobe Systems, which later evolved into an open standard so that multiple applications could create PDF versions of their documents. The purpose of a PDF file is to provide a way of viewing the text and graphics in a document independent of the software that did the original layout. In the early 1990s, the heyday of desktop publishing, each graphic design software package had a different proprietary format for its files, and the packages were quite expensive. In those days, in order to view a document created in Word, Pagemaker, or Quark, you would have to open the document using the same software that had created it. This was especially problematic in the early days of the Web, since there were not many available techniques in HTML to create sophisticated layouts, but people still wanted to share files with each other. PDF was meant to be a vendor-neutral layout format. Adobe made its Acrobat Reader software free for anyone to download, and subsequently the PDF format became widely used. Here is a fun fact about the early days of Acrobat Reader. The words click here when entered into Google search engine still bring up Adobe's Acrobat PDF Reader download website as the first result, and have done so for years. This is because so many websites distribute PDF files along with a message saying something like, "To view this file you must have Acrobat Reader installed. Click here to download it." Since Google's search algorithm uses the link text to learn what sites go with what keywords, the keyword click here is now associated with Adobe Acrobat's download site. PDF is still used to make vendor- and application-neutral versions of files that have layouts that are more complicated than what could be achieved with plain text. For example, viewing the same document in the various versions of Microsoft Word still sometimes causes documents with lots of embedded tables, styles, images, forms, and fonts to look different from one another. This can be due to a number of factors, such as differences in operating systems or versions of the installed Word software itself. Even with applications that are intended to be compatible between software packages or versions, subtle differences can result in incompatibilities. PDF was created to solve some of this. Right away we can tell that PDF is going to be more difficult to deal with than a text file, because it is a binary format, and because it has embedded fonts, images, and so on. So most of the tools in our trusty data cleaning toolbox, such as text editors and command-line tools (less) are largely useless with PDF files. Fortunately there are still a few tricks we can use to get the data out of a PDF file. Try simple solutions first – copying Suppose that on your way to decant your bottle of fine red wine, you spill the bottle on the floor. Your first thought might be that this is a complete disaster and you will have to replace the whole carpet. But before you start ripping out the entire floor, it is probably worth trying to clean the mess with an old bartender's trick: club soda and a damp cloth. In this section, we outline a few things to try first, before getting involved in an expensive file renovation project. They might not work, but they are worth a try. Our experimental file Let's practice cleaning PDF data by using a real PDF file. We also do not want this experiment to be too easy, so let's choose a very complicated file. Suppose we are interested in pulling the data out of a file we found on the Pew Research Center's website called "Is College Worth It?". Published in 2011, this PDF file is 159 pages long and contains numerous data tables showing various ways of measuring if attaining a college education in the United States is worth the investment. We would like to find a way to quickly extract the data within these numerous tables so that we can run some additional statistics on it. For example, here is what one of the tables in the report looks like: This table is fairly complicated. It only has six columns and eight rows, but several of the rows take up two lines, and the header row text is only shown on five of the columns. The complete report can be found at the PewResearch website at http://www.pewsocialtrends.org/2011/05/15/is-college-worth-it/, and the particular file we are using is labeled Complete Report: http://www.pewsocialtrends.org/files/2011/05/higher-ed-report.pdf. Step one – try copying out the data we want The data we will experiment on in this example is found on page 149 of the PDF file (labeled page 143 in their document). If we open the file in a PDF viewer, such as Preview on Mac OSX, and attempt to select just the data in the table, we already see that some strange things are happening. For example, even though we did not mean to select the page number (143); it got selected anyway. This does not bode well for our experiment, but let's continue. Copy the data out by using Command-C or select Edit | Copy. How text looks when selected in this PDF from within Preview Step two – try pasting the copied data into a text editor The following screenshot shows how the copied text looks when it is pasted into Text Wrangler, our text editor: Clearly, this data is not in any sensible order after copying and pasting it. The page number is included, the numbers are horizontal instead of vertical, and the column headers are out of order. Even some of the numbers have been combined; for example, the final row contains the numbers 4,4,3,2; but in the pasted version, this becomes a single number 4432. It would probably take longer to clean up this data manually at this point than it would have taken just to retype the original table. We can conclude that with this particular PDF file, we are going to have to take stronger measures to clean it. Step three – make a smaller version of the file Our copying and pasting procedures have not worked, so we have resigned ourselves to the fact that we are going to need to prepare for more invasive measures. Perhaps if we are not interested in extracting data from all 159 pages of this PDF file, we can identify just the area of the PDF that we want to operate on, and save that section to a separate file. To do this in Preview on MacOSX, launch the File | Print… dialog box. In the Pages area, we will enter the range of pages we actually want to copy. For the purpose of this experiment, we are only interested in page 149; so enter 149 in both the From: and to: boxes as shown in the following screenshot. Then from the PDF dropdown box at the bottom, select Open PDF in Preview. You will see your single-page PDF in a new window. From here, we can save this as a new file and give it a new name, such as report149.pdf or the like. Another technique to try – pdfMiner Now that we have a smaller file to experiment with, let's try some programmatic solutions to extract the text and see if we fare any better. pdfMiner is a Python package with two embedded tools to operate on PDF files. We are particularly interested in experimenting with one of these tools, a command-line program called pdf2txt that is designed to extract text from within a PDF document. Maybe this will be able to help us get those tables of numbers out of the file correctly. Step one – install pdfMiner Launch the Canopy Python environment. From the Canopy Terminal Window, run the following command: pip install pdfminer This will install the entire pdfMiner package and all its associated command-line tools. The documentation for pdfMiner and the two tools that come with it, pdf2txt and dumpPDF, is located at http://www.unixuser.org/~euske/python/pdfminer/. Step two – pull text from the PDF file We can extract all text from a PDF file using the command-line tool called pdf2txt.py. To do this, use the Canopy Terminal and navigate to the directory where the file is located. The basic format of the command is pdf2txt.py <filename>. If you have a larger file that has multiple pages (or you did not already break the PDF into smaller ones), you can also run pdf2txt.py –p149 <filename> to specify that you only want page 149. Just as with the preceding copy-and-paste experiment, we will try this technique not only on the tables located on page 149, but also on the Preface on page 3. To extract just the text from page 3, we run the following command: pdf2txt.py –p3 pewReport.pdf After running this command, the extracted preface of the Pew Research report appears in our command-line window: To save this text to a file called pewPreface.txt, we can simply add a redirect to our command line as follows: pdf2txt.py –p3 pewReport.pdf > pewPreface.txt But what about those troublesome data tables located on page 149? What happens when we use pdf2txt on those? We can run the following command: pdf2txt.py pewReport149.pdf The results are slightly better than copy and paste, but not by much. The actual data output section is shown in the following screenshot. The column headers and data are mixed together, and the data from different columns are shown out of order. We will have to declare the tabular data extraction portion of this experiment a failure, though pdfMiner worked reasonably well on line-by-line text-only extraction. Remember that your success with each of these tools may vary. Much of it depends on the particular characteristics of the original PDF file. It looks like we chose a very tricky PDF for this example, but let's not get disheartened. Instead, we will move on to another tool and see how we fare with it. Third choice – Tabula Tabula is a Java-based program to extract data within tables in PDF files. We will download the Tabula software and put it to work on the tricky tables in our page 149 file. Step one – download Tabula Tabula is available to be downloaded from its website at http://tabula.technology/. The site includes some simple download instructions. On Mac OSX version 10.10.1, I had to download the legacy Java 6 application before I was able to run Tabula. The process was straightforward and required only following the on-screen instructions. Step two – run Tabula Launch Tabula from inside the downloaded .zip archive. On the Mac, the Tabula application file is called simply Tabula.app. You can copy this to your Applications folder if you like. When Tabula starts, it launches a tab or window within your default web browser at the address http://127.0.0.1:8080/. The initial action portion of the screen looks like this: The warning that auto-detecting tables takes a long time is true. For the single-page perResearch149.pdf file, with three tables in it, table auto-detection took two full minutes and resulted in an error message about an incorrectly formatted PDF file. Step three – direct Tabula to extract the data Once Tabula reads in the file, it is time to direct it where the tables are. Using your mouse cursor, select the table you are interested in. I drew a box around the entire first table. Tabula took about 30 seconds to read in the table, and the results are shown as follows: Compared to the way the data was read with copy and paste and pdf2txt, this data looks great. But if you are not happy with the way Tabula reads in the table, you can repeat this process by clearing your selection and redrawing the rectangle. Step four – copy the data out We can use the Download Data button within Tabula to save the data to a friendlier file format, such as CSV or TSV. Step five – more cleaning Open the CSV file in Excel or a text editor and take a look at it. At this stage, we have had a lot of failures in getting this PDF data extracted, so it is very tempting to just quit now. Here are some simple data cleaning tasks: We can combine all the two-line text cells into a single cell. For example, in column B, many of the phrases take up more than one row. Prepare students to be productive and members of the workforce should be in one cell as a single phrase. The same is true for the headers in Rows 1 and 2 (4-year and Private should be in a single cell). To clean this in Excel, create a new column between columns B and C. Use the concatenate() function to join B3:B4, B5:B6, and so on. Use Paste-Special to add the new concatenated values into a new column. Then remove the two columns you no longer need. Do the same for rows 1 and 2. Remove blank lines between rows. When these procedures are finished, the data looks like this: Tabula might seem like a lot of work compared to cutting and pasting data or running a simple command-line tool. That is true, unless your PDF file turns out to be finicky like this one was. Remember that specialty tools are there for a reason—but do not use them unless you really need them. Start with a simple solution first and only proceed to a more difficult tool when you really need it. When all else fails – fourth technique Adobe Systems sells a paid, commercial version of their Acrobat software that has some additional features above and beyond just allowing you to read PDF files. With the full version of Acrobat, you can create complex PDF files and manipulate existing files in various ways. One of the features that is relevant here is the Export Selection As… option found within Acrobat. To get started using this feature, launch Acrobat and use the File Open dialog to open the PDF file. Within the file, navigate to the table holding the data you want to export. The following screenshot shows how to select the data from the page 149 PDF we have been operating on. Use your mouse to select the data, then right-click and choose Export Selection As… At this point, Acrobat will ask you how you want the data exported. CSV is one of the choices. Excel Workbook (.xlsx) would also be a fine choice if you are sure you will not want to also edit the file in a text editor. Since I know that Excel can also open CSV files, I decided to save my file in that format so I would have the most flexibility between editing in Excel and my text editor. After choosing the format for the file, we will be prompted for a filename and location for where to save the file. When we launch the resulting file, either in a text editor or in Excel, we can see that it looks a lot like the Tabula version we saw in the previous section. Here is how our CSV file will look when opened in Excel: At this point, we can use the exact same cleaning routine we used with the Tabula data, where we concatenated the B2:B3 cells into a single cell and then removed the empty rows. Summary The goal of this article was to learn how to export data out of a PDF file. Like sediment in a fine wine, the data in PDF files can appear at first to be very difficult to separate. Unlike decanting wine, however, which is a very passive process, separating PDF data took a lot of trial and error. We learned four ways of working with PDF files to clean data: copying and pasting, pdfMiner, Tabula, and Acrobat export. Each of these tools has certain strengths and weaknesses: Copying and pasting costs nothing and takes very little work, but is not as effective with complicated tables. pdfMiner/Pdf2txt is also free, and as a command-line tool, it could be automated. It also works on large amounts of data. But like copying and pasting, it is easily confused by certain types of tables. Tabula takes some work to set up, and since it is a product undergoing development, it does occasionally give strange warnings. It is also a little slower than the other options. However, its output is very clean, even with complicated tables. Acrobat gives similar output to Tabula, but with almost no setup and very little effort. It is a paid product. By the end, we had a clean dataset that was ready for analysis or long-term storage. Resources for Article: Further resources on this subject: Machine Learning Using Spark MLlib [article] Data visualization [article] First steps with R [article]

(For more resources related to this topic, see here.) Getting ready Before you begin, you will need a running copy of Minecraft: Pi Edition. Start a game in a new or existing world, and wait for the game world to load. How to do it... Follow these steps to connect to the running Minecraft game: Open a fresh terminal by double-clicking on the LXTerminal icon on the desktop. Type cd ~/mcpi/api/python/mcpi into the terminal. Type python to begin the Python interpreter. Enter the following Python code: import minecraftmc = minecraft.Minecraft.create()mc.postToChat("Hello, world!") In the Minecraft window, you should see a message appear! How it works... First, we used the cd command, which we have seen previously to move to the location where the Python API is located. The application programming interface (API) consists of code provided by the Minecraft developers that handles some of the more basic functionality you might need. We used the ~ character as a shortcut for your home directory (/home/pi). Typing in cd /home/pi/mcpi/api/python/mcpi would have exactly the same effect, but requires more typing. We then start the Python interpreter. An interpreter is a program that executes code line by line as it is being typed. This allows us to get instant feedback on the code we are writing. You may like to explore the IDLE interpreter by typing idle into the terminal instead of python. IDLE is more advanced, and is able to color your code based on its meaning (so you can spot errors more easily), and can graphically suggest functions available for use. Then we started writing real Python code. The first line, import minecraft, gives us access to the necessary parts of the API by loading the minecraft module. There are several Python code files inside the directory we moved to, one of which is called minecraft.py, each containing a different code module. The main module we want access to is called minecraft. We then create a connection to the game using mc = minecraft.Minecraft.create(). mc is the name we have given to the connection, which allows us to use the same connection in any future code. minecraft. tells Python to look in the minecraft module. Minecraft is the name of a class in the minecraft module that groups together related data and functions. create() is a function of the Minecraft class that creates a connection to the game. Finally, we use the connection we have created, and its postToChat method to display a message in Minecraft. The way that our code interacts with the game is completely hidden from us to increase fl exibility: we can use almost exactly the same code to interact with any game of Minecraft: Pi Edition, and it is possible to use many different programming languages. If the developers want to change the way the communication works, they can do so, and it won't affect any of the code we have written. Behind the scenes, some text describing our command is sent across a network connection to the game, where the command is interpreted and performed. By default, the connection is to the very Raspberry Pi that we are running the code on, but it is also possible to send these commands over the Internet from any computer to any network-connected Raspberry Pi running Minecraft. A description of all of these text-based messages can be found in ~/ mcpi/api/spec: the message sent to the game when we wrote mc.postToChat("Hello,world!") was chat.post("Hello, world!"). This way of doing things allows any programming language to communicate with the running Minecraft game. As well as Python, a Java API is included that is capable of all the same tasks, and the community has created versions of the API in several other languages. There's more... There are many more functions provided in the Python API, some of the main ones are described here. You can explore the available commands using Python's help function: after importing the minecraft module, help(minecraft) will list the contents of that module, along with any text descriptions provided by the writer of the module. You can also use help to provide information on classes and functions. It is also possible to create your own API by building on top of the existing functions. For example, if you find yourself wanting to create a lot of spheres, you could write your own function that makes use of those provided, and import your module wherever you need it. The minecraft module The following code assumes that you have imported the minecraft module and created a connection to the running Minecraft game using mc = minecraft.Minecraft.create(). Whenever x, y, and z coordinates are used, x and z are both different directions that follow the ground, and y is the height, with 0 at sea level. Code Description mc.getBlock(x,y,z) This gets the type of a block at a particular location as a number. These numbers are all provided in the block module. mc.setBlock(x,y,z, type) The sets the block at a particular position to a particular type. There is also a setBlocks function that allows a cuboid to be filled - this will be faster than setting blocks individually. mc.getHeight(x,z) This gets the height of the world at the given location. mc.getPlayerEntityIds() This gets a list of IDs of all connected players. mc.saveCheckpoint() This saves the current state of the world. mc.restoreCheckpoint() This restores the state of the world from the saved checkpoint. mc.postToChat(message) This posts a message to the game chat. mc.setting(setting, status) This changes a game setting (such as "world_immutable" or "nametags_visible") to True or False. mc.camera.setPos(x,y,z) This moves the game camera to a particular location. Other options are setNormal(player_id), setFixed(), and setFollow(player_id). mc.player.getPos() This gets the position of the host player. mc.player.setPos(x,y,z) This moves the host player. mc.events.pollBlockHits() This gets a list of all blocks that have been hit since the last time the events were requested. Each event describes the position of the block that was hit. mc.events.clearAll() This clears the list of events. At the time of writing, only block hits are recorded, but more event types may be included in the future. The block module Another useful module is the block module: use import block to gain access to its contents. The block module has a list of all available blocks, and the numbers used to represent them. For example, a block of dirt is represented by the number 3. You can use 3 directly in your code if you like, or you can use the helpful name block.DIRT, which will help to make your code more readable. Some blocks, such as wool, have additional information to describe their color. This data can be provided after the block's ID in all functions. For example, to create a block of red wool, where 14 is the data value representing "red": mc.setBlock(x, y, z, block.WOOL, 14) Full information on the additional data values can be found online at http://www.minecraftwiki.net/wiki/Data values(Pocket_Edition). Summary This article gave us a simple code to interact with the game. It also explained how Python communicates with the game. The other aspects brought forth by the article include overview of other API functions, which can build useful functions on top of existing ones. Resources for Article: Further resources on this subject: Creating a file server (Samba) [Article] Webcam and Video Wizardry [Article] Instant Minecraft Designs – Building a Tudor-style house [Article]