Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
All Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

6719 Articles
article-image-how-to-create-a-desktop-application-with-electron-tutorial
Bhagyashree R
06 Feb 2019
15 min read
Save for later

How to create a desktop application with Electron [Tutorial]

Bhagyashree R
06 Feb 2019
15 min read
Electron is an open source framework, created by GitHub, that lets you develop desktop executables that bring together Node and Chrome to provide a full GUI experience. Electron has been used for several well-known projects, including developer tools such as Visual Studio Code, Atom, and Light Table. Basically, you can define the UI with HTML, CSS, and JS (or using React, as we'll be doing), but you can also use all of the packages and functions in Node. So, you won't be limited to a sandboxed experience, being able to go beyond what you could do with just a browser. This article is taken from the book  Modern JavaScript Web Development Cookbook by Federico Kereki.  This problem-solving guide teaches you popular problems solving techniques for JavaScript on servers, browsers, mobile phones, and desktops. To follow along with the examples implemented in this article, you can download the code from the book's GitHub repository. In this article, we will look at how we can use Electron together with the tools like, React and Node, to create a native desktop application, which you can distribute to users. Setting up Electron We will start with installing Electron, and then in the later recipes, we'll see how we can turn a React app into a desktop program. You can install Electron by executing the following command: npm install electron --save-dev Then, we'll need a starter JS file. Taking some tips from the main.js file, we'll create the following electron-start.js file: // Source file: electron-start.js /* @flow */ const { app, BrowserWindow } = require("electron"); let mainWindow; const createWindow = () => { mainWindow = new BrowserWindow({ height: 768, width: 1024 }); mainWindow.loadURL("http://localhost:3000"); mainWindow.on("closed", () => { mainWindow = null; }); }; app.on("ready", createWindow); app.on("activate", () => mainWindow === null && createWindow()); app.on( "window-all-closed", () => process.platform !== "darwin" && app.quit() ); Here are some points to note regarding the preceding code snippet: This code runs in Node, so we are using require() instead of import The mainWindow variable will point to the browser instance where our code will run We'll start by running our React app, so Electron will be able to load the code from http://localhost:3000 In our code, we also have to process the following events: "ready" is called when Electron has finished its initialization and can start creating windows. "closed" means your window was closed; your app might have several windows open, so at this point, you should delete the closed one. "window-all-closed" implies your whole app was closed. In Windows and Linux, this means quitting, but for macOS, you don't usually quit applications, because of Apple' s usual rules. "activate" is called when your app is reactivated, so if the window had been deleted (as in Windows or Linux), you have to create it again. We already have our React app (you can find the React app in the GitHub repository) in place, so we just need a way to call Electron. Add the following script to package.json, and you'll be ready: "scripts": { "electron": "electron .", . . . How it works... To run the Electron app in development mode, we have to do the following: Run our restful_server_cors server code from the GitHub repository. Start the React app, which requires the server to be running. Wait until it's loaded, and then and only then, move on to the next step. Start Electron. So, basically, you'll have to run the following two commands, but you'll need to do so in separate terminals: // in the directory for our restful server: node out/restful_server_cors.js // in the React app directory: npm start // and after the React app is running, in other terminal: npm run electron After starting Electron, a screen quickly comes up, and we again find our countries and regions app, now running independently of a browser: The app works as always; as an example, I selected a country, Canada, and correctly got its list of regions: We are done! You can see that everything is interconnected, as before, in the sense that if you make any changes to the React source code, they will be instantly reflected in the Electron app. Adding Node functionality to your app In the previous recipe, we saw that with just a few small configuration changes, we can turn our web page into an application. However, you're still restricted in terms of what you can do, because you are still using only those features available in a sandboxed browser window. You don't have to think this way, for you can add basically all Node functionality using functions that let you go beyond the limits of the web. Let's see how to do it in this recipe. How to do it We want to add some functionality to our app of the kind that a typical desktop would have. The key to adding Node functions to your app is to use the remote module in Electron. With it, your browser code can invoke methods of the main process, and thus gain access to extra functionality. Let's say we wanted to add the possibility of saving the list of a country's regions to a file. We'd require access to the fs module to be able to write a file, and we'd also need to open a dialog box to select what file to write to. In our serviceApi.js file, we would add the following functions: // Source file: src/regionsApp/serviceApi.js /* @flow */ const electron = window.require("electron").remote; . . . const fs = electron.require("fs"); export const writeFile = fs.writeFile.bind(fs); export const showSaveDialog = electron.dialog.showSaveDialog; Having added this, we can now write files and show dialog boxes from our main code. To use this functionality, we could add a new action to our world.actions.js file: // Source file: src/regionsApp/world.actions.js /* @flow */ import { getCountriesAPI, getRegionsAPI, showSaveDialog, writeFile } from "./serviceApi"; . . . export const saveRegionsToDisk = () => async ( dispatch: ({}) => any, getState: () => { regions: [] } ) => { showSaveDialog((filename: string = "") => { if (filename) { writeFile(filename, JSON.stringify(getState().regions), e => e && window.console.log(`ERROR SAVING ${filename}`, e); ); } }); }; When the saveRegionsToDisk() action is dispatched, it will show a dialog to prompt the user to select what file is to be written, and will then write the current set of regions, taken from getState().regions, to the selected file in JSON format. We just have to add the appropriate button to our <RegionsTable> component to be able to dispatch the necessary action: // Source file: src/regionsApp/regionsTableWithSave.component.js /* @flow */ import React from "react"; import PropTypes from "prop-types"; import "../general.css"; export class RegionsTable extends React.PureComponent<{ loading: boolean, list: Array<{ countryCode: string, regionCode: string, regionName: string }>, saveRegions: () => void }> { static propTypes = { loading: PropTypes.bool.isRequired, list: PropTypes.arrayOf(PropTypes.object).isRequired, saveRegions: PropTypes.func.isRequired }; static defaultProps = { list: [] }; render() { if (this.props.list.length === 0) { return <div className="bordered">No regions.</div>; } else { const ordered = [...this.props.list].sort( (a, b) => (a.regionName < b.regionName ? -1 : 1) ); return ( <div className="bordered"> {ordered.map(x => ( <div key={x.countryCode + "-" + x.regionCode}> {x.regionName} </div> ))} <div> <button onClick={() => this.props.saveRegions()}> Save regions to disk </button> </div> </div> ); } } } We are almost done! When we connect this component to the store, we'll simply add the new action, as follows: // Source file: src/regionsApp/regionsTableWithSave.connected.js /* @flow */ import { connect } from "react-redux"; import { RegionsTable } from "./regionsTableWithSave.component"; import { saveRegionsToDisk } from "./world.actions"; const getProps = state => ({ list: state.regions, loading: state.loadingRegions }); const getDispatch = (dispatch: any) => ({ saveRegions: () => dispatch(saveRegionsToDisk()) }); export const ConnectedRegionsTable = connect( getProps, getDispatch )(RegionsTable); How it works The code we added showed how we could gain access to a Node package (fs, in our case) and some extra functions, such as showing a Save to disk dialog. When we run our updated app and select a country, we'll see our newly added button, as in the following screenshot: Clicking on the button will pop up a dialog, allowing you to select the destination for the data: If you click Save, the list of regions will be written in JSON format, as we specified earlier in our writeRegionsToDisk() function. Building a more windowy experience In the previous recipe, we added the possibility of using any and all of the functions provided by Node. In this recipe, let's now focus on making our app more window-like, with icons, menus, and so on. We want the user to really believe that they're using a native app, with all the features that they would be accustomed to. The following list of interesting subjects from Electron APIs is just a short list of highlights, but there are many more available options: clipboardTo do copy and paste operations using the system's clipboarddialogTo show the native system dialogs for messages, alerts, opening and saving files, and so onglobalShortcutTo detect keyboard shortcutsMenu, MenuItemTo create a menu bar with menus and submenusNotificationTo add desktop notificationspowerMonitor, powerSaveBlockerTo monitor power state changes, and to disable entering sleep modescreenTo get information about the screen, displays, and so onTrayTo add icons and context menus to the system's tray Let's add a few of these functions so that we can get a better-looking app that is more integrated to the desktop. How to do it Any decent app should probably have at least an icon and a menu, possibly with some keyboard shortcuts, so let's add those features now, and just for the sake of it, let's also add some notifications for when regions are written to disk. Together with the Save dialog we already used, this means that our app will include several native windowing features. To start with, let's add an icon. Showing an icon is the simplest thing because it just requires an extra option when creating the BrowserWindow() object. I'm not very graphics-visual-designer oriented, so I just downloaded the Alphabet, letter, r Icon Free file from the Icon-Icons website. Implement the icon as follows: mainWindow = new BrowserWindow({ height: 768, width: 1024, icon: "./src/regionsApp/r_icon.png" }); You can also choose icons for the system tray, although there's no way of using our regions app in that context, but you may want to look into it nonetheless. To continue, the second feature we'll add is a menu, with some global shortcuts to boot. In our App.regions.js file, we'll need to add a few lines to access the Menu module, and to define our menu itself: // Source file: src/App.regions.js . . . import { getRegions } from "./regionsApp/world.actions"; . . . const electron = window.require("electron").remote; const { Menu } = electron; const template = [ { label: "Countries", submenu: [ { label: "Uruguay", accelerator: "Alt+CommandOrControl+U", click: () => store.dispatch(getRegions("UY")) }, { label: "Hungary", accelerator: "Alt+CommandOrControl+H", click: () => store.dispatch(getRegions("HU")) } ] }, { label: "Bye!", role: "quit" } ]; const mainMenu = Menu.buildFromTemplate(template); Menu.setApplicationMenu(mainMenu); Using a template is a simple way to create a menu, but you can also do it manually, adding item by item. I decided to have a Countries menu with two options to show the regions for Uruguay and Hungary. The click property dispatches the appropriate action. I also used the accelerator property to define global shortcuts. See the accelerator.md for the list of possible key combinations to use, including the following: Command keys, such as Command (or Cmd), Control (or Ctrl), or both (CommandOrControl or CmdOrCtrl) Alternate keys, such as Alt, AltGr, or Option Common keys, such as Shift, Escape (or Esc), Tab, Backspace, Insert, or Delete Function keys, such as F1 to F24 Cursor keys, including Up, Down, Left, Right, Home, End, PageUp, and PageDown Media keys, such as MediaPlayPause, MediaStop, MediaNextTrack, MediaPreviousTrack, VolumeUp, VolumeDown, and VolumeMute I also want to be able to quit the application. A complete list of roles is available at Electron docs. With these roles, you can do a huge amount, including some specific macOS functions, along with the following: Work with the clipboard (cut, copy, paste, and pasteAndMatchStyle) Handle the window (minimize, close, quit, reload, and forceReload) Zoom (zoomIn, zoomOut, and resetZoom) To finish, and really just for the sake of it, let's add a notification trigger for when a file is written. Electron has a Notification module, but I opted to use node-notifier, which is quite simple to use. First, we'll add the package in the usual fashion: npm install node-notifier --save In serviceApi.js, we'll have to export the new function, so we'll able to import from elsewhere, as we'll see shortly: const electron = window.require("electron").remote; . . . export const notifier = electron.require("node-notifier"); Finally, let's use this in our world.actions.js file: import { notifier, . . . } from "./serviceApi"; With all our setup, actually sending a notification is quite simple, requiring very little code: // Source file: src/regionsApp/world.actions.js . . . export const saveRegionsToDisk = () => async ( dispatch: ({}) => any, getState: () => { regions: [] } ) => { showSaveDialog((filename: string = "") => { if (filename) { writeFile(filename, JSON.stringify(getState().regions), e => { if (e) { window.console.log(`ERROR SAVING ${filename}`, e); } else { notifier.notify({ title: "Regions app", message: `Regions saved to ${filename}` }); } }); } }); }; How it works First, we can easily check that the icon appears: Now, let's look at the menu. It has our options, including the shortcuts: Then, if we select an option with either the mouse or the global shortcut, the screen correctly loads the expected regions: Finally, let's see if the notifications work as expected. If we click on the Save regions to disk button and select a file, we'll see a notification, as in the following screenshot: Making a distributable package Now that we have a full app, all that's left to do is package it up so that you can deliver it as an executable file for Windows, Linux, or macOS users. How to do it. There are many ways of packaging an app, but we'll use a tool, electron-builder, that will make it even easier, if you can get its configuration right! First of all, we'll have to begin by defining the build configuration, and our initial step will be, as always, to install the tool: npm install electron-builder --save-dev To access the added tool, we'll require a new script, which we'll add in package.json: "scripts": { "dist": "electron-builder", . . . } We'll also have to add a few more details to package.json, which are needed for the build process and the produced app. In particular, the homepage change is required, because the CRA-created index.html file uses absolute paths that won't work later with Electron: "name": "chapter13", "version": "0.1.0", "description": "Regions app for chapter 13", "homepage": "./", "license": "free", "author": "Federico Kereki", Finally, some specific building configuration will be required. You cannot build for macOS with a Linux or Windows machine, so I'll leave that configuration out. We have to specify where the files will be found, what compression method to use, and so on: "build": { "appId": "com.electron.chapter13", "compression": "normal", "asar": true, "extends": null, "files": [ "electron-start.js", "build/**/*", "node_modules/**/*", "src/regionsApp/r_icon.png" ], "linux": { "target": "zip" }, "win": { "target": "portable" } } We have completed the required configuration, but there are also some changes to do in the code itself, and we'll have to adapt the code for building the package. When the packaged app runs, there won't be any webpack server running; the code will be taken from the built React package. The starter code will require the following changes: // Source file: electron-start.for.builder.js /* @flow */ const { app, BrowserWindow } = require("electron"); const path = require("path"); const url = require("url"); let mainWindow; const createWindow = () => { mainWindow = new BrowserWindow({ height: 768, width: 1024, icon: path.join(__dirname, "./build/r_icon.png") }); mainWindow.loadURL( url.format({ pathname: path.join(__dirname, "./build/index.html"), protocol: "file", slashes: true }) ); mainWindow.on("closed", () => { mainWindow = null; }); }; app.on("ready", createWindow); app.on("activate", () => mainWindow === null && createWindow()); app.on( "window-all-closed", () => process.platform !== "darwin" && app.quit() ); Mainly, we are taking icons and code from the build/ directory. An npm run build command will take care of generating that directory, so we can proceed with creating our executable app. How it works After doing this setup, building the app is essentially trivial. Just do the following, and all the distributable files will be found in the dist/ directory: npm run electron-builder Now that we have the Linux app, we can run it by unzipping the .zip file and clicking on the chapter13 executable. (The name came from the "name" attribute in package.json, which we modified earlier.) The result should be like what's shown in the following screenshot: I also wanted to try out the Windows EXE file. Since I didn't have a Windows machine, I made do by downloading a free VirtualBox virtual machine. After downloading the virtual machine, setting it up in VirtualBox, and finally running it, the result that was produced was the same as for Linux: So, we've managed to develop a React app, enhanced it with the Node and Electron features, and finally packaged it for different operating systems. With that, we are done! If you found this post useful, do check out the book, Modern JavaScript Web Development Cookbook.  You will learn how to create native mobile applications for Android and iOS with React Native, build client-side web applications using React and Redux, and much more. How to perform event handling in React [Tutorial] Flutter challenges Electron, soon to release a desktop client to accelerate mobile development Electron 3.0.0 releases with experimental textfield, and button APIs
Read more
  • 0
  • 0
  • 28404

article-image-digital-forensics-using-autopsy
Savia Lobo
24 May 2018
10 min read
Save for later

Getting started with Digital forensics using Autopsy

Savia Lobo
24 May 2018
10 min read
Digital forensics involves the preservation, acquisition, documentation, analysis, and interpretation of evidence from various storage media types. It is not only limited to laptops, desktops, tablets, and mobile devices but also extends to data in transit which is transmitted across public or private networks. In this tutorial, we will cover how one can carry out digital forensics with Autopsy. Autopsy is a digital forensics platform and graphical interface to the sleuth kit and other digital forensics tools. This article is an excerpt taken from the book, 'Digital Forensics with Kali Linux', written by Shiva V.N. Parasram. Let's proceed with the analysis using the Autopsy browser by first getting acquainted with the different ways to start Autopsy. Starting Autopsy Autopsy can be started in two ways. The first uses the Applications menu by clicking on Applications | 11 - Forensics | autopsy: Alternatively, we can click on the Show applications icon (last item in the side menu) and type autopsy into the search bar at the top-middle of the screen and then click on the autopsy icon: Once the autopsy icon is clicked, a new terminal is opened showing the program information along with connection details for opening The Autopsy Forensic Browser. In the following screenshot, we can see that the version number is listed as 2.24 with the path to the Evidence Locker folder as /var/lib/autopsy: To open the Autopsy browser, position the mouse over the link in the terminal, then right-click and choose Open Link, as seen in the following screenshot: Creating a new case To create a new case, follow the given steps: When the Autopsy Forensic Browser opens, investigators are presented with three options. Click on NEW CASE: Enter details for the Case Name, Description, and Investigator Names. For the Case Name, I've entered SP-8-dftt, as it closely matches the image name (8-jpeg-search.dd), which we will be using for this investigation. Once all information is entered, click NEW CASE: Several investigator name fields are available, as there may be instances where several investigators may be working together. The locations of the Case directory and Configuration file are displayed and shown as created.  It's important to take note of the case directory location, as seen in the screenshot: Case directory (/var/lib/autopsy/SP-8-dftt/) created. Click ADD HOST to continue: Enter the details for the Host Name (name of the computer being investigated) and the Description of the host. Optional settings: Time zone: Defaults to local settings, if not specified Timeskew Adjustment: Adds a value in seconds to compensate for time differences Path of Alert Hash Database: Specifies the path of a created database of known bad hashes Path of Ignore Hash Database: Specifies the path of a created database of known good hashes similar to the NIST NSRL: Click on the ADD HOST button to continue. Once the host is added and directories are created, we add the forensic image we want to analyze by clicking the ADD IMAGE button: Click on the ADD IMAGE FILE button to add the image file: To import the image for analysis, the full path must be specified. On my machine, I've saved the image file (8-jpeg-search.dd) to the Desktop folder. As such, the location of the file would be /root/Desktop/ 8-jpeg-search.dd. For the Import Method, we choose Symlink. This way the image file can be imported from its current location (Desktop) to the Evidence Locker without the risks associated with moving or copying the image file. If you are presented with the following error message, ensure that the specified image location is correct and that the forward slash (/) is used: Upon clicking Next, the Image File Details are displayed. To verify the integrity of the file, select the radio button for Calculate the hash value for this image, and select the checkbox next to Verify hash after importing? The File System Details section also shows that the image is of a ntfs partition. Click on the ADD button to continue: After clicking the ADD button in the previous screenshot, Autopsy calculates the MD5 hash and links the image into the evidence locker. Press OK to continue: At this point, we're just about ready to analyze the image file. If there are multiple cases listed in the gallery area from any previous investigations you may have worked on, be sure to choose the 8-jpeg-search.dd file and case: Before proceeding, we can click on the IMAGE DETAILS option. This screen gives detail such as the image name, volume ID, file format, file system, and also allows for the extraction of ASCII, Unicode, and unallocated data to enhance and provide faster keyword searches. Click on the back button in the browser to return to the previous menu and continue with the analysis: Before clicking on the ANALYZE button to start our investigation and analysis, we can also verify the integrity of the image by creating an MD5 hash, by clicking on the IMAGE INTEGRITY button: Several other options exist such as FILE ACTIVITY TIMELINES, HASH DATABASES, and so on. We can return to these at any point in the investigation. After clicking on the IMAGE INTEGRITY button, the image name and hash are displayed. Click on the VALIDATE button to validate the MD5 hash: The validation results are displayed in the lower-left corner of the Autopsy browser window: We can see that our validation was successful, with matching MD5 hashes displayed in the results. Click on the CLOSE button to continue. To begin our analysis, we click on the ANALYZE button: Analysis using Autopsy Now that we've created our case, added host information with appropriate directories, and added our acquired image, we get to the analysis stage. After clicking on the ANALYZE button (see the previous screenshot), we're presented with several options in the form of tabs, with which to begin our investigation: Let's look at the details of the image by clicking on the IMAGE DETAILS tab. In the following snippet, we can see the Volume Serial Number and the operating system (Version) listed as Windows XP: Next, we click on the FILE ANALYSIS tab. This mode opens into File Browsing Mode, which allows the examination of directories and files within the image. Directories within the image are listed by default in the main view area: In File Browsing Mode, directories are listed with the Current Directory specified as C:/. For each directory and file, there are fields showing when the item was WRITTEN, ACCESSED, CHANGED, and CREATED, along with its size and META data: WRITTEN: The date and time the file was last written to ACCESSED: The date and time the file was last accessed (only the date is accurate) CHANGED: The date and time the descriptive data of the file was modified CREATED: The data and time the file was created META: Metadata describing the file and information about the file: For integrity purposes, MD5 hashes of all files can be made by clicking on the GENERATE MD5 LIST OF FILES button. Investigators can also make notes about files, times, anomalies, and so on, by clicking on the ADD NOTE button: The left pane contains four main features that we will be using: Directory Seek: Allows for the searching of directories File Name Search: Allows for the searching of files by Perl expressions or filenames ALL DELETED FILES: Searches the image for deleted files EXPAND DIRECTORIES: Expands all directories for easier viewing of contents By clicking on EXPAND DIRECTORIES, all contents are easily viewable and accessible within the left pane and main window. The + next to a directory indicates that it can be further expanded to view subdirectories (++) and their contents: To view deleted files, we click on the ALL DELETED FILES button in the left pane. Deleted files are marked in red and also adhere to the same format of WRITTEN, ACCESSED, CHANGED, and CREATED times. From the following screenshot, we can see that the image contains two deleted files: We can also view more information about this file by clicking on its META entry. By viewing the metadata entries of a file (last column to the right), we can also view the hexadecimal entries for the file, which may give the true file extensions, even if the extension was changed. In the preceding screenshot, the second deleted file (file7.hmm) has a peculiar file extension of .hmm. Click on the META entry (31-128-3) to view the metadata: Under the Attributes section, click on the first cluster labelled 1066 to view header information of the file: We can see that the first entry is .JFIF, which is an abbreviation for JPEG File Interchange Format. This means that the file7.hmm file is an image file but had its extension changed to .hmm. Sorting files Inspecting the metadata of each file may not be practical with large evidence files. For such an instance, the FILE TYPE feature can be used. This feature allows for the examination of existing (allocated), deleted (unallocated), and hidden files. Click on the FILE TYPE tab to continue: Click Sort files into categories by type (leave the default-checked options as they are) and then click OK to begin the sorting process: Once sorting is complete, a results summary is displayed. In the following snippet, we can see that there are five Extension Mismatches: To view the sorted files, we must manually browse to the location of the output folder, as Autopsy 2.4 does not support viewing of sorted files. To reveal this location, click on View Sorted Files in the left pane: The output folder locations will vary depending on the information specified by the user when first creating the case, but can usually be found at /var/lib/autopsy/<case name>/<host name>/output/sorter-vol#/index.html. Once the index.html file has been opened, click on the Extension Mismatch link: The five listed files with mismatched extensions should be further examined by viewing metadata content, with notes added by the investigator. Reopening cases in Autopsy Cases are usually ongoing and can easily be restarted by starting Autopsy and clicking on OPEN CASE: In the CASE GALLERY, be sure to choose the correct case name and, from there, continue your examination: To recap, we looked at forensics using the Autopsy Forensic Browser with The Sleuth Kit. Compared to individual tools, Autopsy has case management features and supports various types of file analysis, searching, and sorting of allocated, unallocated, and hidden files. Autopsy can also perform hashing on a file and directory levels to maintain evidence integrity. If you enjoyed reading this article, do check out, 'Digital Forensics with Kali Linux' to take your forensic abilities and investigations to a professional level, catering to all aspects of a digital forensic investigation from hashing to reporting. What is Digital Forensics? IoT Forensics: Security in an always connected world where things talk Working with Forensic Evidence Container Recipes
Read more
  • 0
  • 0
  • 27947

article-image-role-angularjs
Packt
16 Dec 2014
7 min read
Save for later

Role of AngularJS

Packt
16 Dec 2014
7 min read
This article by Sandeep Kumar Patel, author of Responsive Web Design with AngularJS we will explore the role of AngularJS for responsive web development. Before going into AngularJS, you will learn about responsive "web development in general. Responsive" web development can be performed "in two ways: Using the browser sniffing approach Using the CSS3 media queries approach (For more resources related to this topic, see here.) Using the browser sniffing approach When we view" web pages through our browser, the browser sends a user agent string to the server. This string" provides information like browser and device details. Reading these details, the browser can be redirected to the appropriate view. This method of reading client details is known as browser sniffing. The browser string has a lot of different information about the source from where the request is generated. The following diagram shows the information shared by the user string:   Details of the parameters" present in the user agent string are as follows: Browser name: This" represents the actual name of the browser from where the request has originated, for example, Mozilla or Opera Browser version: This represents" the browser release version from the vendor, for example, Firefox has the latest version 31 Browser platform: This represents" the underlying engine on which the browser is running, for example, Trident or WebKit Device OS: This represents" the operating system running on the device from where the request has originated, for example, Linux or Windows Device processor: This represents" the processor type on which the operating system is running, for example, 32 or 64 bit A different browser string is generated based on the combination of the device and type of browser used while accessing a web page. The following table shows examples of browser "strings: Browser Device User agent string Firefox Windows desktop Mozilla/5.0 (Windows NT 5.1; rv:31.0) Gecko/20100101 Firefox/31.0 Chrome OS X 10 desktop Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/29.0.1547.66 Safari/537.36 Opera Windows desktop Opera/9.80 (Windows NT 6.0) Presto/2.12.388 Version/12.14 Safari OS X 10 desktop Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.13+ (KHTML, like Gecko) Version/5.1.7 Safari/534.57.2 Internet Explorer Windows desktop Mozilla/5.0 (compatible; MSIE 10.6; Windows NT 6.1; Trident/5.0; InfoPath.2; SLCC1; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 2.0.50727) 3gpp-gba UNTRUSTED/1.0   AngularJS has features like providers or services which can be most useful for this browser user-agent sniffing and a redirection approach. An AngularJS provider can be created that can be used in the configuration in the routing module. This provider can have reusable properties and reusable methods that can be used to identify the device and route the specific request to the appropriate template view. To discover more about user agent strings on various browser and device combinations, visit http://www.useragentstring.com/pages/Browserlist/. CSS3 media queries approach CSS3 brings a "new horizon to web application development. One of the key features" is media queries to develop a responsive web application. Media queries uses media types and features as "deciding parameters to apply the style to the current web page. Media type CSS3 media queries" provide rules for media types to have different styles applied to a web page. In the media queries specification, media types that should be supported by the implemented browser are listed. These media types are as follows: all: This is used" for all media type devices aural: This is "used for speech and sound synthesizers braille: This is used "for braille tactile feedback devices embossed: This" is used for paged braille printers handheld: This is "used for small or handheld devices, for example, mobile print: This" is used for printers, for example, an A4 size paper document projection: This is" used for projection-based devices, such as a projector screen with a slide screen: This is "used for computer screens, for example, desktop and "laptop screens tty: This is" used for media using a fixed-pitch character grid, such as teletypes and terminals tv: This is used "for television-type devices, for example, webOS "or Android-based television A media rule can be declared using the @media keyword with the specific type for the targeted media. The following code shows an example of the media rule usage, where the background body color" is black and text is white for the screen type media, and background body color is white and text is black for the printer media type: @media screen { body {    background:black;    color:white; } }   @media print{ body {    background:white;    color:black; } } An external style "sheet can be downloaded and applied to the current page based on the media type with the HTML link tag. The following code uses the link type in conjunction with media type: <link rel='stylesheet' media='screen' href='<fileName.css>' /> To learn more about" different media types,visit https://developer.mozilla.org/en-US/docs/Web/CSS/@media#Media_types. Media feature Conditional styles can be "applied to a page based on different features of a device. The features that are "supported by CSS3 media queries to apply styles are as follows: color: Based on the" number of bits used for a color component by the device-specific style sheet, this can be applied to a page. color-index: Based "on the color look up, table styles can be applied "to a page. aspect-ratio: Based "on the aspect ratio, display area style sheets can be applied to a page. device-aspect-ratio: Based "on the device aspect ratio, styles can be applied to a page. device-height: Based "on device height, styles can be applied to a page. "This includes the entire screen. device-width: Based "on device width, styles can be applied to a page. "This includes the entire screen. grid: Based "on the device type, bitmap or grid, styles can be applied "to a page. height: Based on" the device rendering area height, styles can be used "to a page. monochrome: Based on" the monochrome type, styles can be applied. "This represents the number of bits used by the device in the grey scale. orientation: Based" on the viewport mode, landscape or portrait, styles can be applied to a page. resolution: Based" on the pixel density, styles can be applied to a page. scan: Based on the "scanning type used by the device for rendering, styles can be applied to a page. width: Based "on the device screen width, specific styles can be applied. The following" code shows some examples" of CSS3 media queries using different device features for conditional styles used: //for screen devices with a minimum aspect ratio 0.5 @media screen and (min-aspect-ratio: 1/2) { img {    height: 70px;    width: 70px; } } //for all device in portrait viewport @media all and (orientation: portrait) { img {    height: 100px;    width: 200px; } } //For printer devices with a minimum resolution of 300dpi pixel density @media print and (min-resolution: 300dpi) { img {    height: 600px;    width: 400px; } } To learn more" about different media features, visit https://developer.mozilla.org/en-US/docs/Web/CSS/@media#Media_features. Summary In this chapter, you learned about responsive design and the SPA architecture. "You now understand the role of the AngularJS library when developing a responsive application. We quickly went through all the important features of AngularJS with the coded syntax. In the next chapter, you will set up your AngularJS application and learn to create dynamic routing-based on the devices. Resources for Article:  Further resources on this subject: Best Practices for Modern Web Applications [article] Important Aspect of AngularJS UI Development [article] A look into responsive design frameworks [article]
Read more
  • 0
  • 20
  • 27911

article-image-getting-started-multiplayer-game-programming
Packt
04 Jun 2015
33 min read
Save for later

Getting Started with Multiplayer Game Programming

Packt
04 Jun 2015
33 min read
In this article by Rodrigo Silveira author of the book Multiplayer gaming with HTML5 game development, if you're reading this, chances are pretty good that you are already a game developer. That being the case, then you already know just how exciting it is to program your own games, either professionally or as a highly gratifying hobby that is very time-consuming. Now you're ready to take your game programming skills to the next level—that is, you're ready to implement multiplayer functionality into your JavaScript-based games. (For more resources related to this topic, see here.) In case you have already set out to create multiplayer games for the Open Web Platform using HTML5 and JavaScript, then you may have already come to realize that a personal desktop computer, laptop, or a mobile device is not particularly the most appropriate device to share with another human player for games in which two or more players share the same game world at the same time. Therefore, what is needed in order to create exciting multiplayer games with JavaScript is some form of networking technology. We will be discussing the following principles and concepts: The basics of networking and network programming paradigms Socket programming with HTML5 Programming a game server and game clients Turn-based multiplayer games Understanding the basics of networking It is said that one cannot program games that make use of networking without first understanding all about the discipline of computer networking and network programming. Although having a deep understanding of any topic can be only beneficial to the person working on that topic, I don't believe that you must know everything there is to know about game networking in order to program some pretty fun and engaging multiplayer games. Saying that is the case is like saying that one needs to be a scholar of the Spanish language in order to cook a simple burrito. Thus, let us take a look at the most basic and fundamental concepts of networking. After you finish reading this article, you will know enough about computer networking to get started, and you will feel comfortable adding multiplayer aspects to your games. One thing to keep in mind is that, even though networked games are not nearly as old as single-player games, computer networking is actually a very old and well-studied subject. Some of the earliest computer network systems date back to the 1950s. Though some of the techniques have improved over the years, the basic idea remains the same: two or more computers are connected together to establish communication between the machines. By communication, I mean data exchange, such as sending messages back and forth between the machines, or one of the machines only sends the data and the other only receives it. With this brief introduction to the concept of networking, you are now grounded in the subject of networking, enough to know what is required to network your games—two or more computers that talk to each other as close to real time as possible. By now, it should be clear how this simple concept makes it possible for us to connect multiple players into the same game world. In essence, we need a way to share the global game data among all the players who are connected to the game session, then continue to update each player about every other player. There are several different techniques that are commonly used to achieve this, but the two most common approaches are peer-to-peer and client-server. Both techniques present different opportunities, including advantages and disadvantages. In general, neither is particularly better than the other, but different situations and use cases may be better suited for one or the other technique. Peer-to-peer networking A simple way to connect players into the same virtual game world is through the peer-to-peer architecture. Although the name might suggest that only two peers ("nodes") are involved, by definition a peer-to-peer network system is one in which two or more nodes are connected directly to each other without a centralized system orchestrating the connection or information exchange. On a typical peer-to-peer setup, each peer serves the same function as every other one—that is, they all consume the same data and share whatever data they produce so that others can stay synchronized. In the case of a peer-to-peer game, we can illustrate this architecture with a simple game of Tic-tac-toe. Once both the players have established a connection between themselves, whoever is starting the game makes a move by marking a cell on the game board. This information is relayed across the wire to the other peer, who is now aware of the decision made by his or her opponent, and can thus update their own game world. Once the second player receives the game's latest state that results from the first player's latest move, the second player is able to make a move of their own by checking some available space on the board. This information is then copied over to the first player who can update their own world and continue the process by making the next desired move. The process goes on until one of the peers disconnects or the game ends as some condition that is based on the game's own business logic is met. In the case of the game of Tic-tac-toe, the game would end once one of the players has marked three spaces on the board forming a straight line or if all nine cells are filled, but neither player managed to connect three cells in a straight path. Some of the benefits of peer-to-peer networked games are as follows: Fast data transmission: Here, the data goes directly to its intended target. In other architectures, the data could go to some centralized node first, then the central node (or the "server") contacts the other peer, sending the necessary updates. Simpler setup: You would only need to think about one instance of your game that, generally speaking, handles its own input, sends its input to other connected peers, and handles their output as input for its own system. This can be especially handy in turn-based games, for example, most board games such as Tic-tac-toe. More reliability: Here one peer that goes offline typically won't affect any of the other peers. However, in the simple case of a two-player game, if one of the players is unable to continue, the game will likely cease to be playable. Imagine, though, that the game in question has dozens or hundreds of connected peers. If a handful of them suddenly lose their Internet connection, the others can continue to play. However, if there is a server that is connecting all the nodes and the server goes down, then none of the other players will know how to talk to each other, and nobody will know what is going on. On the other hand, some of the more obvious drawbacks of peer-to-peer architecture are as follows: Incoming data cannot be trusted: Here, you don't know for sure whether or not the sender modified the data. The data that is input into a game server will also suffer from the same challenge, but once the data is validated and broadcasted to all the other peers, you can be more confident that the data received by each peer from the server will have at least been sanitized and verified, and will be more credible. Fault tolerance can be very low: If enough players share the game world, one or more crashes won't make the game unplayable to the rest of the peers. Now, if we consider the many cases where any of the players that suddenly crash out of the game negatively affect the rest of the players, we can see how a server could easily recover from the crash. Data duplication when broadcasting to other peers: Imagine that your game is a simple 2D side scroller, and many other players are sharing that game world with you. Every time one of the players moves to the right, you receive the new (x, y) coordinates from that player, and you're able to update your own game world. Now, imagine that you move your player to the right by a very few pixels; you would have to send that data out to all of the other nodes in the system. Overall, peer-to-peer is a very powerful networking architecture and is still widely used by many games in the industry. Since current peer-to-peer web technologies are still in their infancy, most JavaScript-powered games today do not make use of peer-to-peer networking. For this and other reasons that should become apparent soon, we will focus almost exclusively on the other popular networking paradigm, namely, the client-server architecture. Client-server networking The idea behind the client-server networking architecture is very simple. If you squint your eyes hard enough, you can almost see a peer-to-peer graph. The most obvious difference between them, is that, instead of every node being an equal peer, one of the nodes is special. That is, instead of every node connecting to every other node, every node (client) connects to a main centralized node called the server. While the concept of a client-server network seems clear enough, perhaps a simple metaphor might make it easier for you to understand the role of each type of node in this network format as well as differentiate it from peer-to-peer . In a peer-to-peer network, you can think of it as a group of friends (peers) having a conversation at a party. They all have access to all the other peers involved in the conversation and can talk to them directly. On the other hand, a client-server network can be viewed as a group of friends having dinner at a restaurant. If a client of the restaurant wishes to order a certain item from the menu, he or she must talk to the waiter, who is the only person in that group of people with access to the desired products and the ability to serve the products to the clients. In short, the server is in charge of providing data and services to one or more clients. In the context of game development, the most common scenario is when two or more clients connect to the same server; the server will keep track of the game as well as the distributed players. Thus, if two players are to exchange information that is only pertinent to the two of them, the communication will go from the first player to and through the server and will end up at the other end with the second player. Following the example of the two players involved in a game of Tic-tac-toe, we can see how similar the flow of events is on a client-server model. Again, the main difference is that players are unaware of each other and only know what the server tells them. While you can very easily mimic a peer-to-peer model by using a server to merely connect the two players, most often the server is used much more actively than that. There are two ways to engage the server in a networked game, namely in an authoritative and a non-authoritative way. That is to say, you can have the enforcement of the game's logic strictly in the server, or you can have the clients handle the game logic, input validation, and so on. Today, most games using the client-server architecture actually use a hybrid of the two (authoritative and non-authoritative servers). For all intents and purposes, however, the server's purpose in life is to receive input from each of the clients and distribute that input throughout the pool of connected clients. Now, regardless of whether you decide to go with an authoritative server instead of a non-authoritative one, you will notice that one of challenges with a client-server game is that you will need to program both ends of the stack. You will have to do this even if your clients do nothing more than take input from the user, forward it to the server, and render whatever data they receive from the server; if your game server does nothing more than forward the input that it receives from each client to every other client, you will still need to write a game client and a game server. We will discuss game clients and servers later. For now, all we really need to know is that these two components are what set this networking model apart from peer-to-peer. Some of the benefits of client-server networked games are as follows: Separation of concerns: If you know anything about software development, you know that this is something you should always aim for. That is, good, maintainable software is written as discrete components where each does one "thing", and it is done well. Writing individual specialized components lets you focus on performing one individual task at a time, making your game easier to design, code, test, reason, and maintain. Centralization: While this can be argued against as well as in favor of, having one central place through which all communication must flow makes it easier to manage such communication, enforce any required rules, control access, and so forth. Less work for the client: Instead of having a client (peer) in charge of taking input from the user as well as other peers, validating all the input, sharing data among other peers, rendering the game, and so on, the client can focus on only doing a few of these things, allowing the server to offload some of this work. This is particularly handy when we talk about mobile gaming, and how much subtle divisions of labor can impact the overall player experience. For example, imagine a game where 10 players are engaged in the same game world. In a peer-to-peer setup, every time one player takes an action, he or she would need to send that action to nine other players (in other words, there would need to be nine network calls, boiling down to more mobile data usage). On the other hand, on a client-server configuration, one player would only need to send his or her action to one of the peers, that is, the server, who would then be responsible for sending that data to the remaining nine players. Common drawbacks of client-server architectures, whether or not the server is authoritative, are as follows: Communication takes longer to propagate: In the very best possible scenario imaginable, every message sent from the first player to the second player would take twice as long to be delivered as compared to a peer-to-peer connection. That is, the message would be first sent from the first player to the server and then from the server to the second player. There are many techniques that are used today to solve the latency problem faced in this scenario, some of which we will discuss in much more depth later. However, the underlying dilemma will always be there. More complexity due to more moving parts: It doesn't really matter how you slice the pizza; the more code you need to write (and trust me, when you build two separate modules for a game, you will write more code), the greater your mental model will have to be. While much of your code can be reused between the client and the server (especially if you use well-established programming techniques, such as object-oriented programming), at the end of the day, you need to manage a greater level of complexity. Single point of failure and network congestion: Up until now, we have mostly discussed the case where only a handful of players participates in the same game. However, the more common case is that a handful of groups of players play different games at the same time. Using the same example of the two-player game of Tic-tac-toe, imagine that there are thousands of players facing each other in single games. In a peer-to-peer setup, once a couple of players have directly paired off, it is as though there are no other players enjoying that game. The only thing to keep these two players from continuing their game is their own connection with each other. On the other hand, if the same thousands of players are connected to each other through a server sitting between the two, then two singled out players might notice severe delays between messages because the server is so busy handling all of the messages from and to all of the other people playing isolated games. Worse yet, these two players now need to worry about maintaining their own connection with each other through the server, but they also hope that the server's connection between them and their opponent will remain active. All in all, many of the challenges involved in client-server networking are well studied and understood, and many of the problems you're likely to face during your multiplayer game development will already have been solved by someone else. Client-server is a very popular and powerful game networking model, and the required technology for it, which is available to us through HTML5 and JavaScript, is well developed and widely supported. Networking protocols – UDP and TCP By discussing some of the ways in which your players can talk to each other across some form of network, we have yet only skimmed over how that communication is actually done. Let us then describe what protocols are and how they apply to networking and, more importantly, multiplayer game development. The word protocol can be defined as a set of conventions or a detailed plan of a procedure [Citation [Def. 3,4]. (n.d.). In Merriam Webster Online, Retrieved February 12, 2015, from http://www.merriam-webster.com/dictionary/protocol]. In computer networking, a protocol describes to the receiver of a message how the data is organized so that it can be decoded. For example, imagine that you have a multiplayer beat 'em up game, and you want to tell the game server that your player just issued a kick command and moved 3 units to the left. What exactly do you send to the server? Do you send a string with a value of "kick", followed by the number 3? Otherwise, do you send the number first, followed by a capitalized letter "K", indicating that the action taken was a kick? The point I'm trying to make is that, without a well-understood and agreed-upon protocol, it is impossible to successfully and predictably communicate with another computer. The two networking protocols that we'll discuss in the section, and that are also the two most widely used protocols in multiplayer networked games, are the Transmission Control Protocol (TCP) and the User Datagram Protocol (UDP). Both protocols provide communication services between clients in a network system. In simple terms, they are protocols that allow us to send and receive packets of data in such a way that the data can be identified and interpreted in a predictable way. When data is sent through TCP, the application running in the source machine first establishes a connection with the destination machine. Once a connection has been established, data is transmitted in packets in such a way that the receiving application can then put the data back together in the appropriate order. TCP also provides built-in error checking mechanisms so that, if a packet is lost, the target application can notify the sender application, and any missing packets are sent again until the entire message is received. In short, TCP is a connection-based protocol that guarantees the delivery of the full data in the correct order. Use cases where this behavior is desirable are all around us. When you download a game from a web server, for example, you want to make sure that the data comes in correctly. You want to be sure that your game assets will be properly and completely downloaded before your users start playing your game. While this guarantee of delivery may sound very reassuring, it can also be thought of as a slow process, which, as we'll see briefly, may sometimes be more important than knowing that the data will arrive in full. In contrast, UDP transmits packets of data (called datagrams) without the use of a pre-established connection. The main goal of the protocol is to be a very fast and frictionless way of sending data towards some target application. In essence, you can think of UDP as the brave employees who dress up as their company's mascot and stand outside their store waving a large banner in the hope that at least some of the people driving by will see them and give them their business. While at first, UDP may seem like a reckless protocol, the use cases that make UDP so desirable and effective includes the many situations when you care more about speed than missing packets a few times, getting duplicate packets, or getting them out of order. You may also want to choose UDP over TCP when you don't care about the reply from the receiver. With TCP, whether or not you need some form of confirmation or reply from the receiver of your message, it will still take the time to reply back to you, at least acknowledging that the message was received. Sometimes, you may not care whether or not the server received the data. A more concrete example of a scenario where UDP is a far better choice over TCP is when you need a heartbeat from the client letting the server know if the player is still there. If you need to let your server know that the session is still active every so often, and you don't care if one of the heartbeats get lost every now and again, then it would be wise to use UDP. In short, for any data that is not mission-critical and you can afford to lose, UDP might be the best option. In closing, keep in mind that, just as peer-to-peer and client-server models can be built side by side, and in the same way your game server can be a hybrid of authoritative and non-authoritative, there is absolutely no reason why your multiplayer games should only use TCP or UDP. Use whichever protocol a particular situation calls for. Network sockets There is one other protocol that we'll cover very briefly, but only so that you can see the need for network sockets in game development. As a JavaScript programmer, you are doubtlessly familiar with Hypertext Transfer Protocol (HTTP). This is the protocol in the application layer that web browsers use to fetch your games from a Web server. While HTTP is a great protocol to reliably retrieve documents from web servers, it was not designed to be used in real-time games; therefore, it is not ideal for this purpose. The way HTTP works is very simple: a client sends a request to a server, which then returns a response back to the client. The response includes a completion status code, indicating to the client that the request is either in process, needs to be forwarded to another address, or is finished successfully or erroneously. There are a handful of things to note about HTTP that will make it clear that a better protocol is needed for real-time communication between the client and server. Firstly, after each response is received by the requester, the connection is closed. Thus, before making each and every request, a new connection must be established with the server. Most of the time, an HTTP request will be sent through TCP, which, as we've seen, can be slow, relatively speaking. Secondly, HTTP is by design a stateless protocol. This means that, every time you request a resource from a server, the server has no idea who you are and what is the context of the request. (It doesn't know whether this is your first request ever or if you're a frequent requester.) A common solution to this problem is to include a unique string with every HTTP request that the server keeps track of, and can thus provide information about each individual client on an ongoing basis. You may recognize this as a standard session. The major downside with this solution, at least with regard to real-time gaming, is that mapping a session cookie to the user's session takes additional time. Finally, the major factor that makes HTTP unsuitable for multiplayer game programming is that the communication is one way—only the client can connect to the server, and the server replies back through the same connection. In other words, the game client can tell the game server that a punch command has been entered by the user, but the game server cannot pass that information along to other clients. Think of it like a vending machine. As a client of the machine, we can request specific items that we wish to buy. We formalize this request by inserting money into the vending machine, and then we press the appropriate button. Under no circumstance will a vending machine issue commands to a person standing nearby. That would be like waiting for a vending machine to dispense food, expecting people to deposit the money inside it afterwards. The answer to this lack of functionality in HTTP is pretty straightforward. A network socket is an endpoint in a connection that allows for two-way communication between the client and the server. Think of it more like a telephone call, rather than a vending machine. During a telephone call, either party can say whatever they want at any given time. Most importantly, the connection between both parties remains open throughout the duration of the conversation, making the communication process highly efficient. WebSocket is a protocol built on top of TCP, allowing web-based applications to have two-way communication with a server. The way a WebSocket is created consists of several steps, including a protocol upgrade from HTTP to WebSocket. Thankfully, all of the heavy lifting is done behind the scenes by the browser and JavaScript. For now, the key takeaway here is that with a TCP socket (yes, there are other types of socket including UDP sockets), we can reliably communicate with a server, and the server can talk back to us as per the need. Socket programming in JavaScript Let's now bring the conversation about network connections, protocols, and sockets to a close by talking about the tools—JavaScript and WebSockets—that bring everything together, allowing us to program awesome multiplayer games in the language of the open Web. The WebSocket protocol Modern browsers and other JavaScript runtime environments have implemented the WebSocket protocol in JavaScript. Don't make the mistake of thinking that just because we can create WebSocket objects in JavaScript, WebSockets are part of JavaScript. The standard that defines the WebSocket protocol is language-agnostic and can be implemented in any programming language. Thus, before you start to deploy your JavaScript games that make use of WebSockets, ensure that the environment that will run your game uses an implementation of the ECMA standard that also implements WebSockets. In other words, not all browsers will know what to do when you ask for a WebSocket connection. For the most part, though, the latest versions, as of this writing, of the most popular browsers today (namely, Google Chrome, Safari, Mozilla Firefox, Opera, and Internet Explorer) implement the current latest revision of RFC 6455. Previous versions of WebSockets (such as protocol version - 76, 7, or 10) are slowly being deprecated and have been removed by some of the previously mentioned browsers. Probably the most confusing thing about the WebSocket protocol is the way each version of the protocol is named. The very first draft (which dates back to 2010), was named draft-hixie-thewebsocketprotocol-75. The next version was named draft-hixie-thewebsocketprotocol-76. Some people refer to these versions as 75 and 76, which can be quite confusing, especially since the fourth version of the protocol is named draft-ietf-hybi-thewebsocketprotocol-07, which is named in the draft as WebSocket Version 7. The current version of the protocol (RFC 6455) is 13. Let us take a quick look at the programming interface (API) that we'll use within our JavaScript code to interact with a WebSocket server. Keep in mind that we'll need to write both the JavaScript clients that use WebSockets to consume data as well as the WebSocket server, which uses WebSockets but plays the role of the server. The difference between the two will become apparent as we go over some examples. Creating a client-side WebSocket The following code snippet creates a new object of type WebSocket that connects the client to some backend server. The constructor takes two parameters; the first is required and represents the URL where the WebSocket server is running and expecting connections. The second URL, is an optional list of sub-protocols that the server may implement. var socket = new WebSocket('ws://www.game-domain.com'); Although this one line of code may seem simple and harmless enough, here are a few things to keep in mind: We are no longer in HTTP territory. The address to your WebSocket server now starts with ws:// instead of http://. Similarly, when we work with secure (encrypted) sockets, we would specify the server's URL as wss://, just like in https://. It may seem obvious to you, but a common pitfall that those getting started with WebSockets fall into is that, before you can establish a connection with the previous code, you need a WebSocket server running at that domain. WebSockets implement the same-origin security model. As you may have already seen with other HTML5 features, the same-origin policy states that you can only access a resource through JavaScript if both the client and the server are in the same domain. For those who are not familiar with the same-domain (also known as the same-origin) policy, the three things that constitute a domain, in this context, are the protocol, host, and port of the resource being accessed. In the previous example, the protocol, host, and port number were, respectively ws (and not wss, http, or ssh), www.game-domain.com (any sub-domain, such as game-domain.com or beta.game-domain.com would violate the same-origin policy), and 80 (by default, WebSocket connects to port 80, and port 443 when it uses wss). Since the server in the previous example binds to port 80, we don't need to explicitly specify the port number. However, had the server been configured to run on a different port, say 2667, then the URL string would need to include a colon followed by the port number that would need to be placed at the end of the host name, such as ws://www.game-domain.com:2667. As with everything else in JavaScript, WebSocket instances attempt to connect to the backend server asynchronously. Thus, you should not attempt to issue commands on your newly created socket until you're sure that the server has connected; otherwise, JavaScript will throw an error that may crash your entire game. This can be done by registering a callback function on the socket's onopen event as follows: var socket = new WebSocket('ws://www.game-domain.com'); socket.onopen = function(event) {    // socket ready to send and receive data }; Once the socket is ready to send and receive data, you can send messages to the server by calling the socket object's send method, which takes a string as the message to be sent. // Assuming a connection was previously established socket.send('Hello, WebSocket world!'); Most often, however, you will want to send more meaningful data to the server, such as objects, arrays, and other data structures that have more meaning on their own. In these cases, we can simply serialize our data as JSON strings. var player = {    nickname: 'Juju',    team: 'Blue' };   socket.send(JSON.stringify(player)); Now, the server can receive that message and work with it as the same object structure that the client sent it, by running it through the parse method of the JSON object. var player = JSON.parse(event.data); player.name === 'Juju'; // true player.team === 'Blue'; // true player.id === undefined; // true If you look at the previous example closely, you will notice that we extract the message that is sent through the socket from the data attribute of some event object. Where did that event object come from, you ask? Good question! The way we receive messages from the socket is the same on both the client and server sides of the socket. We must simply register a callback function on the socket's onmessage event, and the callback will be invoked whenever a new message is received. The argument passed into the callback function will contain an attribute named data, which will contain the raw string object with the message that was sent. socket.onmessage = function(event) {    event instanceof MessageEvent; // true      var msg = JSON.parse(event.data); }; Other events on the socket object on which you can register callbacks include onerror, which is triggered whenever an error related to the socket occurs, and onclose, which is triggered whenever the state of the socket changes to CLOSED; in other words, whenever the server closes the connection with the client for any reason or the connected client closes its connection. As mentioned previously, the socket object will also have a property called readyState, which behaves in a similar manner to the equally-named attribute in AJAX objects (or more appropriately, XMLHttpRequest objects). This attribute represents the current state of the connection and can have one of four values at any point in time. This value is an unsigned integer between 0 and 3, inclusive of both the numbers. For clarity, there are four accompanying constants on the WebSocket class that map to the four numerical values of the instance's readyState attribute. The constants are as follows: WebSocket.CONNECTING: This has a value of 0 and means that the connection between the client and the server has not yet been established. WebSocket.OPEN: This has a value of 1 and means that the connection between the client and the server is open and ready for use. Whenever the object's readyState attribute changes from CONNECTING to OPEN, which will only happen once in the object's life cycle, the onopen callback will be invoked. WebSocket.CLOSING: This has a value of 2 and means that the connection is being closed. WebSocket.CLOSED: This has a value of 3 and means that the connection is now closed (or could not be opened to begin with). Once the readyState has changed to a new value, it will never return to a previous state in the same instance of the socket object. Thus, if a socket object is CLOSING or has already become CLOSED, it will never OPEN again. In this case, you would need a new instance of WebSocket if you would like to continue to communicate with the server. To summarize, let us bring together the simple WebSocket API features that we discussed previously and create a convenient function that simplifies data serialization, error checking, and error handling when communicating with the game server: function sendMsg(socket, data) {    if (socket.readyState === WebSocket.OPEN) {      socket.send(JSON.stringify(data));        return true;    }      return false; }; Game clients Earlier, we talked about the architecture of a multiplayer game that was based on the client-server pattern. Since this is the approach we will take for the games that we'll be developing, let us define some of the main roles that the game client will fulfill. From a higher level, a game client will be the interface between the human player and the rest of the game universe (which includes the game server and other human players who are connected to it). Thus, the game client will be in charge of taking input from the player, communicating this to the server, receive any further instructions and information from the server, and then render the final output to the human player again. Depending on the type of game server used, the client can be more sophisticated than just an input application that renders static data received from the server. For example, the client could very well simulate what the game server will do and present the result of this simulation to the user while the server performs the real calculations and tells the results to the client. The biggest selling point of this technique is that the game would seem a lot more dynamic and real-time to the user since the client responds to input almost instantly. Game servers The game server is primarily responsible for connecting all the players to the same game world and keeping the communication going between them. However as you will soon realize, there may be cases where you will want the server to be more sophisticated than a routing application. For example, just because one of the players is telling the server to inform the other participants that the game is over, and the player sending the message is the winner, we may still want to confirm the information before deciding that the game is in fact over. With this idea in mind, we can label the game server as being of one of the two kinds: authoritative or non-authoritative. In an authoritative game server, the game's logic is actually running in memory (although it normally doesn't render any graphical output like the game clients certainly will) all the time. As each client reports the information to the server by sending messages through its corresponding socket, the server updates the current game state and sends the updates back to all of the players, including the original sender. This way we can be more certain that any data coming from the server has been verified and is accurate. In a non-authoritative server, the clients take on a much more involved part in the game logic enforcement, which gives the client a lot more trust. As suggested previously, what we can do is take the best of both worlds and create a mix of both the techniques. What we will do is, have a strictly authoritative server, but clients that are smart and can do some of the work on their own. Since the server has the ultimate say in the game, however, any messages received by clients from the server are considered as the ultimate truth and supersede any conclusions it came to on its own. Summary Overall, we discussed the basics of networking and network programming paradigms. We saw how WebSockets makes it possible to develop real-time, multiplayer games in HTML5. Finally, we implemented a simple game client and game server using widely supported web technologies and built a fun game of Tic-tac-toe. Resources for Article: Further resources on this subject: HTML5 Game Development – A Ball-shooting Machine with Physics Engine [article] Creating different font files and using web fonts [article] HTML5 Canvas [article]
Read more
  • 0
  • 0
  • 27886

article-image-creating-simple-gamemanager-using-unity3d
Ellison Leao
09 Jan 2015
5 min read
Save for later

Creating a simple GameManager using Unity3D

Ellison Leao
09 Jan 2015
5 min read
Using the so called "Game Managers" in games is just as common as eating when making games. Probably every game made has their natural flow: Start -> Play -> Pause -> Die -> Game Over , etc. To handle these different game states, we need a proper manager who can provide a mechanism to know when to change to state "A" to state "B" during gameplay. In this post we will show you how to create a simple game manager for Unity3D games. We will assume that you have some previous knowledge in Unity, but if you haven't get the chance to know it, please go to the Official Learn Unity page and get started. We are going to create the scripts using the C# language. 1 - The Singleton Pattern For the implementation, we will use the Singleton pattern. Why? Some reasons: One instance for all the game implementation, with no possible duplications. The instance is never destroyed on scene changes. It stores the current game state to be accessible anytime. We will not explain the design of the Singleton pattern because it's not the purpose of this post. If you wish to know more about it, you can go here. 2 - The GameManager code Create a new project on Unity and add a first csharp script called SimpleGameManager.cs and add the following code: using UnityEngine; using System.Collections; // Game States // for now we are only using these two public enum GameState { INTRO, MAIN_MENU } public delegate void OnStateChangeHandler(); public class SimpleGameManager { protected SimpleGameManager() {} private static SimpleGameManager instance = null; public event OnStateChangeHandler OnStateChange; public GameState gameState { get; private set; } public static SimpleGameManager Instance{ get { if (SimpleGameManager.instance == null){ DontDestroyOnLoad(SimpleGameManager.instance); SimpleGameManager.instance = new SimpleGameManager(); } return SimpleGameManager.instance; } } public void SetGameState(GameState state){ this.gameState = state; OnStateChange(); } public void OnApplicationQuit(){ SimpleGameManager.instance = null; } } Explaining the code in parts, we have: First we are making some enums for easily check the Game State, so for this example we will have: public enum GameState { INTRO, MAIN_MENU } Then we will have an event delegate method that we will use as a callback when a game state changes. This is ideal for changing scenes. public delegate void OnStateChangeHandler(); Moving forward we will have the gameState attribute, that is a getter for the current Game State. public GameState gameState {get; private set;} Then we will have our class. Taking a look at the singleton implementation we can see that we will use the Instance static variable to get our Game Manager current instance or create a new one if it doesn't exists. It's also interesting to see that we call the DontDestroyOnLoad method in the Game Manager instanciation. On doing that, Unity makes sure that our instance is never destroyed between scenes. The method used to change the Game State is SetGameState, which we only need to pass the GameState enum variable as the parameter. public void SetGameState(GameState state){ this.gameState = state; OnStateChange(); } It automatically sets the new gameState for the instance and call the callback OnStateChangemethod. 3 - Creating Sample Scenes For testing our new Game Manager, we will create 2 Unity scenes: Intro and Menu. The Intro scene will just show some debug messages, simulating an Intro game scene, and after 3 seconds it will change to the Menu Scene were we have the Game Menu code. Create a new scene called Intro and create a csharp script called Intro.cs. Put the following code into the script: using UnityEngine; using System.Collections; public class Intro : MonoBehaviour { SimpleGameManager GM; void Awake () { GM = SimpleGameManager.Instance; GM.OnStateChange += HandleOnStateChange; Debug.Log("Current game state when Awakes: " + GM.gameState); } void Start () { Debug.Log("Current game state when Starts: " + GM.gameState); } public void HandleOnStateChange () { GM.SetGameState(GameState.MAIN_MENU); Debug.Log("Handling state change to: " + GM.gameState); Invoke("LoadLevel", 3f); } public void LoadLevel(){ Application.LoadLevel("Menu"); } } You can see here that we just need to call the Game Manager instance inside the Awake method. The same initialization will happen on the others scripts, to get the current Game Manager state. After getting the Game Manager instance we set the OnStateChange event, which is load the Menu scene after 3 seconds. You can notice that the first line of the event sets the new Game State by calling the SetGameState method. If you run this scene however, you will get an error because we don't have the Menu.cs Scene yet. So let's create it! Create a new scene called Menu and add a csharp script called Menu.cs into this Scene. Add the following code to Menu.cs: using UnityEngine; using System.Collections; public class Menu : MonoBehaviour { SimpleGameManager GM; void Awake () { GM = SimpleGameManager.Instance; GM.OnStateChange += HandleOnStateChange; } public void HandleOnStateChange () { Debug.Log("OnStateChange!"); } public void OnGUI(){ //menu layout GUI.BeginGroup (new Rect (Screen.width / 2 - 50, Screen.height / 2 - 50, 100, 800)); GUI.Box (new Rect (0, 0, 100, 200), "Menu"); if (GUI.Button (new Rect (10, 40, 80, 30), "Start")){ StartGame(); } if (GUI.Button (new Rect (10, 160, 80, 30), "Quit")){ Quit(); } GUI.EndGroup(); } public void StartGame(){ //start game scene GM.SetGameState(GameState.GAME); Debug.Log(GM.gameState); } public void Quit(){ Debug.Log("Quit!"); Application.Quit(); } } We added simple Unity GUI elements for this scene just for example. Run the Intro Scene and check the Debug logs, You should see the messages when the Game State is changing from the old state to the new state and keeping the instance between scenes. And there you have it! You can add more GameStates for multiple screens like Credits, High Score, Levels, etc. The code for this examples is on github, feel free to fork and use it in your games! https://github.com/bttfgames/SimpleGameManager About this Author  Ellison Leão (@ellisonleao) is a passionate software engineer with more than 6 years of experience in web projects and contributor to the MelonJS framework and other open source projects. When he is not writing games, he loves to play drums.
Read more
  • 0
  • 1
  • 27819

article-image-how-setup-postgresql-nodejs
Antonio Cucciniello
14 Feb 2017
7 min read
Save for later

How to Setup PostgreSQL with Node.js

Antonio Cucciniello
14 Feb 2017
7 min read
Have you ever wanted to add a PostgreSQL database to the backend of your web application? If so, by the end of this tutorial, you should have a PostgreSQL database up and running with your Node.js web application. PostgreSQL is a popular open source relational database. This tutorial assumes that you have Node and NPM installed on your machine; if you need help installing that, check out this link. First, let's download PostgreSQL. PostgreSQL I am writing and testing this tutorial on a Mac, so it will primarily caterMac, but I will include links in the reference section for downloading PostgreSQL on select Linux distributions and Windows as well. If you are on a Mac, however, you can follow these steps. First, you must have Homebrew. If you do not have it, you may install it byfollowing the directions here. Once Homebrew is installed and working, you can run the following: $ brew update $ brew install postgres This downloads and installs PostgreSQL for you. Command-line Setup Now, open a new instance of a terminal by pressing Command+T. Once you have the new tab, you can start a PostgreSQL server with the command: postgres -D /usr/local/var/postgres.This allows you to use Postgres locally and gives you a logger for all of the commands you run on your databases. Next, open a new instance of a terminal with Command+Tand enter$ psql. This is similar to a command center for Postgres. It allows you to create things in your database and plenty more. You can manually enter commands here to set up your environment. For this example, we willcreatea database called example. To do that, while in the terminal tab with psql running, enter CREATE DATABASE example;. To confirm that the database was made, you should seeCREATE DATABASE as the output. Also, to list all databases, you would usel. Then, we will want to connect to our new database with the command connect example. This should give you the following message telling you that you are connected: You are connected to database "example" as user In order to store things in this database, we need to create a table. Enter the following: CREATE TABLE numbers(   age integer   ); So, this format is probably confusing if you have never seen it before. This is telling Postgres to create a table in this database called numbers, with one column called age, and all items in the age column will be of the data type integer. Now, this should give us the output CREATE TABLE,but if you want to list all tables in a database, you shoulduse the dt command. For the sake of this example, we are going to add a row to the table so that we have some data to play with and prove that this works. When you want to add something to a database in PostgreSQL, you use the INSERT command. Enter this command to have the first row in the table equal to 732: INSERT INTO numbers VALUES (732); This should give you an output of INSERT 0 1. To check the contents of the table, simply typeTABLE numbers;. Now that we have a database up and running with a table with a value, we can setup our code to access this table and pull the value from it. Code Setup In order to follow this example, you will need to make sure that you have the following packages: pg, pg-format, and express. Enter the project directory you plan on working in (and where you have Node and NPMinstalled). For pg, usenpm install -pg.This is a Postgres client for Node. For pg-format, usenpm install pg-format.This allows us to safely make dynamic SQL queries. For express,use npm install express --save.This allows us to create a quick and basic server. Now that those packages are installed, we can code! Actual Code Let's create a file called app.js for this as the main point in our program. At the top, establish your variables: const express = require('express') const app = express() var pg = require('pg') var format = require('pg-format') var PGUSER = 'yourUserName' var PGDATABASE = 'example' var age = 732 The first two lines allow us to use the package express and help us make our server. The next two lines allow us to use the packages pg and pg-format. PGUSER is a variable that holds the user to your database. Enter your username here in place of yourUserName. PGDATABASE is a variable to hold the database name that we wouldlike to connect to. Then, the last variable is to hold the number that we stored in the database. Next, add this: var config = {   user: PGUSER, // name of the user account   database: PGDATABASE, // name of the database   max: 10, // max number of clients in the pool   idleTimeoutMillis: 30000 // how long a client is allowed to remain idle before being closed } var pool = new pg.Pool(config) var myClient Here, we establish a config object that allows pg to know that we want to connect to the database specified as the user specified, with a maximum of 10 clients in a pool of clients with a time out of 30,000 milliseconds of how long a client can be idle before disconnected from the database. Then, we create a new pool of clients according to that config file. Afterwards, we create a variable called myClient to store the client we get from the database in the next step. Now, enter the last bit of code here: pool.connect(function (err, client, done) { if (err) console.log(err) app.listen(3000, function () { console.log('listening on 3000') }) myClient = client var ageQuery = format('SELECT * from numbers WHERE age = %L', age) myClient.query(ageQuery, function (err, result) { if (err) { console.log(err) } console.log(result.rows[0]) }) }) This tries to connect to the database with one of the clients from the pool. If a client successfully connects to the database, we start our server by listening on a port (here, I use 3000). Then, we get access to our client. Next,we create a variable called ageQuery to make a dynamic SQL query. A query is a command to a database. Here, we are making a SELECT query to the database, checking all rows in the table called numbers where the age column is equal to 732. If that is a successful query (meaning, it finds a row with 732 as the value), then we will log the answer. It's now time to test all your hard work! Save the file and run the command in a terminal: node app.js Your output should look like this: listening on 3000 { age: 732 } Conclusion There you go! You now have a PostgreSQL database connected to your web app. To summarize our work, here is a quick breakdown of what happened: We installed PostgreSQL through Homebrew. We started our Local PostgreSQL server. We opened psql in a terminal to use commands manually. We created a database called example. We created a table in that database called numbers. We added a value to that table. We installed pg, pg-format, and express. We used Express to create a server. We created a pool of clients using a config object to access the database. We queried the table in the database for 732. We logged the value. Check out the code for this tutorial on GitHub. About the Author Antonio Cucciniello is a software engineer with a background in C, C++, and Javascript (Node.Js) from New Jersey. His most recent project called Edit Docs is an Amazon Echo skill that allows users to edit Google Drive files using our voice. He loves building cool things with software and reading books on self-help and improvement, finance, and entrepreneurship. You can find Antonio on Twitter @antocucciniello and on GitHub.
Read more
  • 0
  • 1
  • 27354
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at £15.99/month. Cancel anytime
article-image-how-to-attack-an-infrastructure-using-voip-exploitation-tutorial
Savia Lobo
03 Nov 2018
9 min read
Save for later

How to attack an infrastructure using VoIP exploitation [Tutorial]

Savia Lobo
03 Nov 2018
9 min read
Voice over IP (VoIP) is pushing business communications to a new level of efficiency and productivity. VoIP-based systems are facing security risks on a daily basis. Although a lot of companies are focusing on the VoIP quality of service, they ignore the security aspects of the VoIP infrastructure, which makes them vulnerable to dangerous attacks. This tutorial is an extract taken from the book Advanced Infrastructure Penetration Testing written by Chiheb Chebbi. In this book, you will explore exploitation abilities such as offensive PowerShell tools and techniques, CI servers, database exploitation, Active Directory delegation, and much more. In today's post, you will learn how to penetrate the VoIP infrastructure. Like any other penetration testing, to exploit the VoIP infrastructure, we need to follow a strategic operation based on a number of steps. Before attacking any infrastructure, we've learned that we need to perform footprinting, scanning, and enumeration before exploiting it, and that is exactly what we are going to do with VoIP. To perform VoIP information gathering, we need to collect as much useful information as possible about the target. As a start, you can do a simple search online. For example, job announcements could be a valuable source of information. For example, the following job description gives the attacker an idea about the VoIP: Later, an attacker could search for vulnerabilities out there to try exploiting that particular system. Searching for phone numbers could also be a smart move, to have an idea of the target based on its voicemail, because each vendor has a default one. If the administrator has not changed it, listening to the voicemail can let you know about your target. If you want to have a look at some of the default voicemails, check http://www.hackingvoip.com/voicemail.html. It is a great resource for learning a great deal about hacking VoIP. Google hacking is an amazing technique for searching for information and online portals. We discussed Google hacking using Dorks. The following demonstration is the output of this Google Dork—in  URL: Network Configuration Cisco: You can find connected VoIP devices using the Shodan.io search engine: VoIP devices are generally connected to the internet. Thus, they can be reached by an outsider. They can be exposed via their web interfaces; that is why, sometimes leaving installation files exposed could be dangerous, because using a search engine can lead to indexing the portal. The following screenshot is taken from an online Asterisk management portal: And this screenshot is taken from a configuration page of an exposed website, using a simple search engine query: After collecting juicy information about the target, from an attacker perspective, we usually should perform scanning. Using scanning techniques is necessary during this phase. Carrying out Host Discovery and Nmap scanning is a good way of scanning the infrastructure to search for VoIP devices. Scanning can lead us to discover VoIP services. For example, we saw the -sV option in Nmap to check services. In VoIP, if port 2000 is open, it is a Cisco CallManager because the SCCP protocol uses that port as default, or if there is a UDP 5060 port, it is SIP. The -O Nmap option could be useful for identifying the running operating system, as there are a lot of VoIP devices that are running on a specific operating system, such as Cisco embedded. You know what to do now. After footprinting and scanning, we need to enumerate the target. As you can see, when exploiting an infrastructure we generally follow the same methodological steps. Banner grabbing is a well-known technique in enumeration, and the first step to enumerate a VoIP infrastructure is by starting a banner grabbing move. In order to do that, using the Netcat utility would help you grab the banner easily, or you can simply use the Nmap script named banner: nmap -sV --script=banner <target> For a specific vendor, there are a lot of enumeration tools you can use; EnumIAX is one of them. It is a built-in enumeration tool in Kali Linux to brute force Inter-Asterisk Exchange protocol usernames: Automated Corporate Enumerator (ACE) is another built-in enumeration tool in Kali Linux: svmap is an open source built-in tool in Kali Linux for identifying SIP devices. Type svmap -h and you will get all the available options for this amazing tool: VoIP attacks By now, you have learned the required skills to perform VoIP footprinting, scanning, and enumeration. Let's discover the major VoIP attacks. VoIP is facing multiple threats from different attack vectors. Denial-of-Service Denial-of-Service (DoS) is a threat to the availability of a network. DoS could be dangerous too for VoIP, as ensuring the availability of calls is vital in modern organizations. Not only the availability but also the clearness of calls is a necessity nowadays. To monitor the QoS of VoIP, you can use many tools that are out there; one of them is CiscoWorks QoS Policy Manager 4.1: To measure the quality of VoIP, there are some scoring systems, such as the Mean Opinion Score (MOS)  or the R-value based on several parameters (jitter, latency, and packet loss). Scores of the mean opinion score range from 1 to 5 (bad to very clear) and scores of R-value range from 1 to 100 (bad to very clear). The following screenshot is taken from an analysis of an RTP packet downloaded from the Wireshark website: You can also analyze the RTP jitter graph: VoIP infrastructure can be attacked by the classic DoS attacks. We saw some of them previously: Smurf flooding attack TCP SYN flood attack UDP flooding attack One of the DoS attack tools is iaxflood. It is available in Kali Linux to perform DoS attacks. IAX stands for  Inter-Asterisk Exchange. Open a Kali terminal and type  iaxflood <Source IP> <Destination IP>  <Number of packets>: The VoIP infrastructure can not only be attacked by the previous attacks attackers can perform packet Fragmentation and Malformed Packets to attack the infrastructure, using fuzzing tools. Eavesdropping Eavesdropping is one of the most serious VoIP attacks. It lets attackers take over your privacy, including your calls. There are many eavesdropping techniques; for example, an attacker can sniff the network for TFTP configuration files while they contain a password. The following screenshot describes an analysis of a TFTP capture: Also, an attacker can harvest phone numbers and build a valid phone numbers databases, after recording all the outgoing and ongoing calls. Eavesdropping does not stop there, attackers can record your calls and even know what you are typing using the Dual-Tone Multi-Frequency (DTMF). You can use the DTMF decoder/encoder from this link http://www.polar-electric.com/DTMF/: Voice Over Misconfigured Internet Telephones (VOMIT) is a great utility to convert Cisco IP Phone conversations into WAV files. You can download it from its official website http://vomit.xtdnet.nl/: SIP attacks Another attacking technique is SIP rogues. We can perform two types of SIP rogues. From an attacker's perspective, we can implement the following: Rogue SIP B2BUA: In  this attacking technique, the attacker mimics SIP B2BUA: SIP rogue as a proxy: Here, the attacker mimics a SIP proxy:   SIP registration hijacking SIP registration hijacking is a serious VoIP security problem. Previously, we saw that before establishing a SIP session, there is a registration step. Registration can be hijacked by attackers. During a SIP registration hijacking attack, the attacker disables a normal user by a Denial of Service, for example, and simply sends a registration request with his own IP address instead of that users because, in SIP, messages are transferred clearly, so SIP does not ensure the integrity of signalling messages: If you are a Metasploit enthusiast, you can try many other SIP modules. Open a Metasploit console by typing msfconsole and search SIP modules using search SIP: To use a specific SIP module, simply type use <module >. The following interface is an example of SIP module usage: Spam over Internet Telephony Spam over Internet Telephony (SPIT), sometimes called Voice spam, is like email spam, but it affects VoIP. To perform a SPIT attack, you can use a generation tool called spitter. Embedding malware Malware is a major threat to VoIP infrastructure. Your insecure VoIP endpoints can be exploited by different types of malware, such as Worms and VoIP Botnets. Softphones are also a highly probable target for attackers. Compromising your softphone could be very dangerous because if an attacker exploits it, they can compromise your VoIP network. Malware is not the only threat against VoIP endpoints. VoIP firmware is a potential attack vector for hackers. Firmware hacking can lead to phones being compromised. Viproy – VoIP penetration testing kit Viproy VoIP penetration testing kit (v4)  is a VoIP and unified communications services pentesting tool presented at Black Hat Arsenal USA 2014 by Fatih Ozavci: To download this project, clone it from its official repository, https://github.com/fozavci/viproy-voipkit: # git clone https://github.com/fozavci/viproy-voipkit. The following project contains many modules to test SIP and Skinny protocols: To use them, copy the lib, modules, and data folders to a Metasploit folder in your system. Thus, in  this article, we demonstrated how to exploit the VoIP infrastructure. We explored the major VoIP attacks and how to defend against them, in addition to the tools and utilities most commonly used by penetration testers. If you've enjoyed reading this, do check out Advanced Infrastructure Penetration Testing to discover post-exploitation tips, tools, and methodologies to help your organization build an intelligent security system. Managing a VoIP Solution with Active Directory Depends On Your Needs Opus 1.3, a popular FOSS audio codec with machine learning and VR support, is now generally available Approaching a Penetration Test Using Metasploit
Read more
  • 0
  • 0
  • 27305

article-image-what-is-a-convolutional-neural-network-cnn-video
Richard Gall
25 Sep 2018
5 min read
Save for later

What is a convolutional neural network (CNN)? [Video]

Richard Gall
25 Sep 2018
5 min read
What is a convolutional neural network, exactly? Well, let's start with the basics: a convolutional neural network (CNN) is a type of neural network that is most often applied to image processing problems. You've probably seen them in action anywhere a computer is identifying objects in an image. But you can also use convolutional neural networks in natural language processing projects, too. The fact that they are useful for these fast growing areas is one of the main reasons they're so important in deep learning and artificial intelligence today. What makes a convolutional neural network unique? Once you understand how a convolutional neural network works and what makes it unique from other neural networks, you can see why they're so effective for processing and classifying images. But let’s first take a regular neural network. A regular neural network has an input layer, hidden layers and an output layer. The input layer accepts inputs in different forms, while the hidden layers perform calculations on these inputs. The output layer then delivers the outcome of the calculations and extractions. Each of these layers contains neurons that are connected to neurons in the previous layer, and each neuron has its own weight. This means you aren’t making any assumptions about the data being fed into the network - great usually, but not if you’re working with images or language. Convolutional neural networks work differently as they treat data as spatial. Instead of neurons being connected to every neuron in the previous layer, they are instead only connected to neurons close to it and all have the same weight. This simplification in the connections means the network upholds the spatial aspect of the data set. It means your network doesn’t think an eye is all over the image. The word ‘convolutional’ refers to the filtering process that happens in this type of network. Think of it this way, an image is complex - a convolutional neural network simplifies it so it can be better processed and ‘understood.’ What's inside a convolutional neural network? Like a normal neural network, a convolutional neural network is made up of multiple layers. There are a couple of layers that make it unique - the convolutional layer and the pooling layer. However, like other neural networks, it will also have a ReLu or rectified linear unit layer, and a fully connected layer. The ReLu layer acts as an activation function, ensuring non-linearity as the data moves through each layer in the network - without it, the data being fed into each layer would lose the dimensionality that we want to maintain. The fully connected layer, meanwhile, allows you to perform classification on your dataset. The convolutional layer The convolutional layer is the most important, so let’s start there. It works by placing a filter over an array of image pixels - this then creates what’s called a convolved feature map. "It’s a bit like looking at an image through a window which allows you to identify specific features you might not otherwise be able to see. The pooling layer Next we have the pooling layer - this downsamples or reduces the sample size of a particular feature map. This also makes processing much faster as it reduces the number of parameters the network needs to process. The output of this is a pooled feature map. There are two ways of doing this, max pooling, which takes the maximum input of a particular convolved feature, or average pooling, which simply takes the average. These steps amount to feature extraction, whereby the network builds up a picture of the image data according to its own mathematical rules. If you want to perform classification, you'll need to move into the fully connected layer. To do this, you'll need to flatten things out - remember, a neural network with a more complex set of connections can only process linear data. How to train a convolutional neural network There are a number of ways you can train a convolutional neural network. If you’re working with unlabelled data, you can use unsupervised learning methods. One of the best popular ways of doing this is using auto-encoders - this allows you to squeeze data in a space with low dimensions, performing calculations in the first part of the convolutional neural network. Once this is done you’ll then need to reconstruct with additional layers that upsample the data you have. Another option is to use generative adversarial networks, or GANs. With a GAN, you train two networks. The first gives you artificial data samples that should resemble data in the training set, while the second is a ‘discriminative network’ - it should distinguish between the artificial and the 'true' model. What's the difference between a convolutional neural network and a recurrent neural network? Although there's a lot of confusion about the difference between a convolutional neural network and a recurrent neural network, it's actually more simple than many people realise. Whereas a convolutional neural network is a feedforward network that filters spatial data, a recurrent neural network, as the name implies, feeds data back into itself. From this perspective recurrent neural networks are better suited to sequential data. Think of it like this: a convolutional network is able to perceive patterns across space - a recurrent neural network can see them over time. How to get started with convolutional neural networks If you want to get started with convolutional neural networks Python and TensorFlow are great tools to begin with. It’s worth exploring MNIST dataset too. This is a database of handwritten digits that you can use to get started with building your first convolutional neural network. To learn more about convolutional neural networks, artificial intelligence, and deep learning, visit Packt's store for eBooks and videos.
Read more
  • 0
  • 0
  • 27202

article-image-using-gerrit-github
Packt
04 Sep 2013
14 min read
Save for later

Using Gerrit with GitHub

Packt
04 Sep 2013
14 min read
In this article by Luca Milanesio, author of the book Learning Gerrit Code review, we will learn about Gerrit Code revew. GitHub is the world's largest platform for the free hosting of Git Projects, with over 4.5 million registered developers. We will now provide a step-by-step example of how to connect Gerrit to an external GitHub server so as to share the same set of repositories. Additionally, we will provide guidance on how to use the Gerrit Code Review workflow and GitHub concurrently. By the end of this article we will have our Gerrit installation fully integrated and ready to be used for both open source public projects and private projects on GitHub. (For more resources related to this topic, see here.) GitHub workflow GitHub has become the most popular website for open source projects, thanks to the migration of some major projects to Git (for example, Eclipse) and new projects adopting it, along with the introduction of the social aspect of software projects that piggybacks on the Facebook hype. The following diagram shows the GitHub collaboration model: The key aspects of the GitHub workflow are as follows: Each developer pushes to their own repository and pulls from others Developers who want to make a change to another repository, create a fork on GitHub and work on their own clone When forked repositories are ready to be merged, pull requests are sent to the original repository maintainer The pull requests include all of the proposed changes and their associated discussion threads Whenever a pull request is accepted, the change is merged by the maintainer and pushed to their repository on GitHub   GitHub controversy The preceding workflow works very effectively for most open source projects; however, when the projects gets bigger and more complex, the tools provided by GitHub are too unstructured, and a more defined review process with proper tools, additional security, and governance is needed. In May 2012 Linus Torvalds , the inventor of Git version control, openly criticized GitHub as a commit editing tool directly on the pull request discussion thread: " I consider GitHub useless for these kinds of things. It's fine for hosting, but the pull requests and the online commit editing, are just pure garbage " and additionally, " the way you can clone a (code repository), make changes on the web, and write total crap commit messages, without GitHub in any way making sure that the end result looks good. " See https://github.com/torvalds/linux/pull/17#issuecomment-5654674. Gerrit provides the additional value that Linus Torvalds claimed was missing in the GitHub workflow: Gerrit and GitHub together allows the open source development community to reuse the extended hosting reach and social integration of GitHub with the power of governance of the Gerrit review engine. GitHub authentication The list of authentication backends supported by Gerrit does not include GitHub and it cannot be used out of the box, as it does not support OpenID authentication. However, a GitHub plugin for Gerrit has been recently released in order to fill the gaps and allow a seamless integration. GitHub implements OAuth 2.0 for allowing external applications, such as Gerrit, to integrate using a three-step browser-based authentication. Using this scheme, a user can leverage their existing GitHub account without the need to provision and manage a separate one in Gerrit. Additionally, the Gerrit instance will be able to self-provision the SSH public keys needed for pushing changes for review. In order for us to use GitHub OAuth authentication with Gerrit, we need to do the following: Build the Gerrit GitHub plugin Install the GitHub OAuth filter into the Gerrit libraries (/lib under the Gerrit site directory) Reconfigure Gerrit to use the HTTP authentication type   Building the GitHub plugin The Gerrit GitHub plugin can be found under the Gerrit plugins/github repository on https://gerrit-review.googlesource.com/#/admin/projects/plugins/github. It is open source under the Apache 2.0 license and can be cloned and built using the Java 6 JDK and Maven. Refer to the following example: $ git clone https://gerrit.googlesource.com/plugins/github $ cd github $ mvn install […] [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------- [INFO] Total time: 9.591s [INFO] Finished at: Wed Jun 19 18:38:44 BST 2013 [INFO] Final Memory: 12M/145M [INFO] ------------------------------------------------------- The Maven build should generate the following artifacts: github-oauth/target/github-oauth*.jar, the GitHub OAuth library for authenticating Gerrit users github-plugin/target/github-plugin*.jar, the Gerrit plugin for integrating with GitHub repositories and pull requests Installing GitHub OAuth library The GitHub OAuth JAR file needs to copied to the Gerrit /lib directory; this is required to allow Gerrit to use it for filtering all HTTP requests and enforcing the GitHub three-step authentication process: $ cp github-oauth/target/github-oauth-*.jar /opt/gerrit/lib/ Installing GitHub plugin The GitHub plugin includes the additional support for the overall configuration, the advanced GitHub repositories replication, and the integration of pull requests into the Code Review process. We now need to install the plugin before running the Gerrit init again so that we can benefit from the simplified automatic configuration steps: $ cp github-plugin/target/github-plugin-*.jar /opt/gerrit/plugins/github.jar Register Gerrit as a GitHub OAuth application Before going through the Gerrit init, we need to tell GitHub to trust Gerrit as a partner application. This is done through the generation of a ClientId/ClientSecret pair associated to the exact Gerrit URLs that will be used for initiating the 3-step OAuth authentication. We can register a new application in GitHub through the URL https://github.com/settings/applications/new, where the following three fields are requested: Application name : It is the logical name of the application authorized to access GitHub, for example, Gerrit. Main URL : The Gerrit canonical web URL used for redirecting to GitHub OAuth authentication, for example, https://myhost.mydomain:8443. Callback URL : The URL that GitHub should redirect to when the OAuth authentication is successfully completed, for example, https://myhost.mydomain:8443/oauth. GitHub will automatically generate a unique pair ClientId/ClientSecret that has to be provided to Gerrit identifying them as a trusted authentication partner. ClientId/ClientSecret are not GitHub credentials and cannot be used by an interactive user to access any GitHub data or information. They are only used for authorizing the integration between a Gerrit instance and GitHub. Running Gerrit init to configure GitHub OAuth We now need to stop Gerrit and go through the init steps again in order to reconfigure the Gerrit authentication. We need to enable HTTP authentication by choosing an HTTP header to be used to verify the user's credentials, and to go through the GitHub settings wizard to configure the OAuth authentication. $ /opt/gerrit/bin/gerrit.sh stop Stopping Gerrit Code Review: OK $ cd /opt/gerrit $ java -jar gerrit.war init [...] *** User Authentication *** Authentication method []: HTTP RETURN Get username from custom HTTP header [Y/n]? Y RETURN Username HTTP header []: GITHUB_USER RETURN SSO logout URL : /oauth/reset RETURN *** GitHub Integration *** GitHub URL [https://github.com]: RETURN Use GitHub for Gerrit login ? [Y/n]? Y RETURN ClientId []: 384cbe2e8d98192f9799 RETURN ClientSecret []: f82c3f9b3802666f2adcc4 RETURN Initialized /opt/gerrit $ /opt/gerrit/bin/gerrit.sh start Starting Gerrit Code Review: OK   Using GitHub login for Gerrit Gerrit is now fully configured to register and authenticate users through GitHub OAuth. When opening the browser to access any Gerrit web pages, we are automatically redirected to the GitHub for login. If we have already visited and authenticated with GitHub previously, the browser cookie will be automatically recognized and used for the authentication, instead of presenting the GitHub login page. Alternatively, if we do not yet have a GitHub account, we create a new GitHub profile by clicking on the SignUp button. Once the authentication process is successfully completed, GitHub requests the user's authorization to grant access to their public profile information. The following screenshot shows GitHub OAuth authorization for Gerrit: The authorization status is then stored under the user's GitHub applications preferences on https://github.com/settings/applications. Finally, GitHub redirects back to Gerrit propagating the user's profile securely using a one-time code which is used to retrieve the full data profile including username, full name, e-mail, and associated SSH public keys. Replication to GitHub The next steps in the Gerrit to GitHub integration is to share the same Git repositories and then keep them up-to-date; this can easily be achieved by using the Gerrit replication plugin. The standard Gerrit replication is a master-slave, where Gerrit always plays the role of the master node and pushes to remote slaves. We will refer to this scheme as push replication because the actual control of the action is given to Gerrit through a git push operation of new commits and branches. Configure Gerrit replication plugin In order to configure push replication we need to enable the Gerrit replication plugin through Gerrit init: $ /opt/gerrit/bin/gerrit.sh stop Stopping Gerrit Code Review: OK $ cd /opt/gerrit $ java -jar gerrit.war init [...] *** Plugins *** Prompt to install core plugins [y/N]? y RETURN Install plugin reviewnotes version 2.7-rc4 [y/N]? RETURN Install plugin commit-message-length-validator version 2.7-rc4 [y/N]? RETURN Install plugin replication version 2.6-rc3 [y/N]? y RETURN Initialized /opt/gerrit $ /opt/gerrit/bin/gerrit.sh start Starting Gerrit Code Review: OK The Gerrit replication plugin relies on the replication.config file under the /opt/gerrit/etc directory to identify the list of target Git repositories to push to. The configuration syntax is a standard .ini format where each group section represents a target replica slave. See the following simplest replication.config script for replicating to GitHub: [remote "github"] url = [email protected]:myorganisation/${name}.git The preceding configuration enables all of the repositories in Gerrit to be replicated to GitHub under the myorganisa tion GitHub Team account. Authorizing Gerrit to push to GitHub Now, that Gerrit knows where to push, we need GitHub to authorize the write operations to its repositories. To do so, we need to upload the SSH public key of the underlying OS user where Gerrit is running to one of the accounts in the GitHub myorganisation team, with the permissions to push to any of the GitHub repositories. Assuming that Gerrit runs under the OS user gerrit, we can copy and paste the SSH public key values from the ~gerrit/.ssh/id_rsa.pub (or ~gerrit/.ssh/id_dsa.pub) to the Add an SSH Key section of the GitHub account under target URL to be set to: https://github.com/settings/ssh Start working with Gerrit replication Everything is now ready to start playing with Gerrit to GitHub replication. Whenever a change to a repository is made on Gerrit, it will be automatically replicated to the corresponding GitHub repository. In reality there is one additional operation that is needed on the GitHub side: the actual creation of the empty repositories using https://github.com/new associated to the ones created in Gerrit. We need to make sure that we select the organization name and repository name, consistent with the ones defined in Gerrit and in the replication.config file. Never initialize the repository from GitHub with an empty commit or readme file; otherwise the first replication attempt from Gerrit will result in a conflict and will then fail. Now GitHub and Gerrit are fully connected and whenever a repository in GitHub matches one of the repositories in Gerrit, it will be linked and synchronized with the latest set of commits pushed in Gerrit. Thanks to the Gerrit-GitHub authentication previously configured, Gerrit and GitHub share the same set of users and the commits authors will be automatically recognized and formatted by GitHub. The following screenshot shows Gerrit commits replicated to GitHub: Reviewing and merging to GitHub branches The final goal of the Code Review process is to agree and merge changes to their branches. The merging strategies need to be aligned with real-life scenarios that may arise when using Gerrit and GitHub concurrently. During the Code Review process the alignment between Gerrit and GitHub was at the change level, not influenced by the evolution of their target branches. Gerrit changes and GitHub pull requests are isolated branches managed by their review lifecycle. When a change is merged, it needs to align with the latest status of its target branch using a fast-forward, merge, rebase, or cherry-pick strategy. Using the standard Gerrit merge functionality, we can apply the configured project merge strategy to the current status of the target branch on Gerrit. The situation on GitHub may have changed as well, so even if the Gerrit merge has succeeded there is no guarantee that the actual subsequent synchronization to GitHub will do the same! The GitHub plugin mitigates this risk by implementing a two-phase submit + merge operation for merging opened changes as follows: Phase-1 : The change target branch is checked against its remote peer on GitHub and fast forwarded if needed. If two branches diverge, the submit + merge is aborted and manual merge intervention is requested. Phase-2 : The change is merged on its target branch in Gerrit and an additional ad hoc replication is triggered. If the merge succeeds then the GitHub pull request is marked as completed. At the end of Phase-2 the Gerrit and GitHub statuses will be completely aligned. The pull request author will then receive the notification that his/her commit has been merged. Using Gerrit and GitHub on http://gerrithub.io When using Gerrit and GitHub on the web with public or private repositories, all of the commits are replicated from Gerrit to GitHub, and each one of them has a complete copy of the data. If we are using a Git and collaboration server on GitHub over the Internet, why can't we do the same for its Gerrit counterpart? Can we avoid installing a standalone instance of Gerrit just for the purpose of going through a formal Code Review? One hassle-free solution is to use the GerritHub service (http://gerrithub.io), which offers a free Gerrit instance on the cloud already configured and connected with GitHub through the github-plugin and github-oauth authentication library. All of the flows that we have covered in this article are completely automated, including the replication and automatic pull request to change automation. As accounts are shared with GitHub, we do not need to register or create another account to use GerritHub; we can just visit http://gerrithub.io and start using Gerrit Code Review with our existing GitHub projects without having to teach our existing community about a new tool. GerritHub also includes an initial setup Wizard for the configuration and automation of the Gerrit projects and the option to configure the Gerrit groups using the existing GitHub. Once Gerrit is configured, the Code Review and GitHub can be used seamlessly for achieving maximum control and social reach within your developer community. Summary We have now integrated our Gerrit installation with GitHub authentication for a seamless Single-Sign-On experience. Using an existing GitHub account we started using Gerrit replication to automatically mirror all the commits to GitHub repositories, allowing our projects to have an extended reach to external users, free to fork our repositories, and to contribute changes as pull requests. Finally, we have completed our Code Review in Gerrit and managed the merge to GitHub with a two-phase change submit + merge process to ensure that the target branches on both Gerrit and GitHub have been merged and aligned accordingly. Similarly to GitHub, this Gerrit setup can be leveraged for free on the web without having to manage a separate private instance, thanks to the free set target URL to http://gerrithub.io service available on the cloud. Resources for Article : Further resources on this subject: Getting Dynamics NAV 2013 on Your Computer – For (Almost) Free [Article] Building Your First Zend Framework Application [Article] Quick start - your first Sinatra application [Article]
Read more
  • 0
  • 1
  • 27106

article-image-fingerprint-detection-using-opencv
Packt
07 Oct 2015
11 min read
Save for later

Fingerprint detection using OpenCV 3

Packt
07 Oct 2015
11 min read
In this article by Joseph Howse, Quan Hua, Steven Puttemans, and Utkarsh Sinha, the authors of OpenCV Blueprints, we delve into the aspect of fingerprint detection using OpenCV. (For more resources related to this topic, see here.) Fingerprint identification, how is it done? We have already discussed the use of the first biometric, which is the face of the person trying to login to the system. However since we mentioned that using a single biometric can be quite risky, we suggest adding secondary biometric checks to the system, like the fingerprint of a person. There are many of the shelf fingerprint scanners which are quite cheap and return you the scanned image. However you will still have to write your own registration software for these scanners and which can be done by using OpenCV software. Examples of such fingerprint images can be found below. Examples of single individual thumb fingerprint in different scanning positions This dataset can be downloaded from the FVC2002 competition website released by the University of Bologna. The website (http://bias.csr.unibo.it/fvc2002/databases.asp) contains 4 databases of fingerprints available for public download of the following structure: Four fingerprint capturing devices DB1 - DB4. For each device, the prints of 10 individuals are available. For each person, 8 different positions of prints were recorded. We will use this publicly available dataset to build our system upon. We will focus on the first capturing device, using up to 4 fingerprints of each individual for training the system and making an average descriptor of the fingerprint. Then we will use the other 4 fingerprints to evaluate our system and make sure that the person is still recognized by our system. You could apply exactly the same approach on the data grabbed from the other devices if you would like to investigate the difference between a system that captures almost binary images and one that captures grayscale images. However we will provide techniques for doing the binarization yourself. Implement the approach in OpenCV 3 The complete fingerprint software for processing fingerprints derived from a fingerprint scanner can be found at https://github.com/OpenCVBlueprints/OpenCVBlueprints/tree/master/chapter_6/source_code/fingerprint/fingerprint_process/. In this subsection we will describe how you can implement this approach in the OpenCV interface. We will start by grabbing the image from the fingerprint system and apply binarization. This will enable us to remove any desired noise from the image as well as help us to make the contrast better between the kin and the wrinkled surface of the finger. // Start by reading in an image Mat input = imread("/data/fingerprints/image1.png", CV_LOAD_GRAYSCALE); // Binarize the image, through local thresholding Mat input_binary; threshold(input, input_binary, 0, 255, CV_THRESH_BINARY | CV_THRESH_OTSU); The Otsu thresholding will automatically choose the best generic threshold for the image to obtain a good contrast between foreground and background information. This is because the image contains a bimodal distribution (which means that we have an image with a 2 peak histogram) of pixel values. For that image, we can approximately take a value in the middle of those peaks as threshold value. (for images which are not bimodal, binarization won't be accurate.) Otsu allows us to avoid using a fixed threshold value, and thus making the system more general for any capturing device. However we do acknowledge that if you have only a single capturing device, then playing around with a fixed threshold value could result in a better image for that specific setup. The result of the thresholding can be seen below. In order to make thinning from the next skeletization step as effective as possible we need the inverse binary image. Comparison between grayscale and binarized fingerprint image Once we have a binary image we are actually already set to go to calculate our feature points and feature point descriptors. However, in order to improve the process a bit more, we suggest to skeletize the image. This will create more unique and stronger interest points. The following piece of code can apply the skeletization on top of the binary image. The skeletization is based on the Zhang-Suen line thinning approach. Special thanks to @bsdNoobz of the OpenCV Q&A forum who supplied this iteration approach. #include <opencv2/imgproc.hpp> #include <opencv2/highgui.hpp> using namespace std; using namespace cv; // Perform a single thinning iteration, which is repeated until the skeletization is finalized void thinningIteration(Mat& im, int iter) { Mat marker = Mat::zeros(im.size(), CV_8UC1); for (int i = 1; i < im.rows-1; i++) { for (int j = 1; j < im.cols-1; j++) { uchar p2 = im.at<uchar>(i-1, j); uchar p3 = im.at<uchar>(i-1, j+1); uchar p4 = im.at<uchar>(i, j+1); uchar p5 = im.at<uchar>(i+1, j+1); uchar p6 = im.at<uchar>(i+1, j); uchar p7 = im.at<uchar>(i+1, j-1); uchar p8 = im.at<uchar>(i, j-1); uchar p9 = im.at<uchar>(i-1, j-1); int A = (p2 == 0 && p3 == 1) + (p3 == 0 && p4 == 1) + (p4 == 0 && p5 == 1) + (p5 == 0 && p6 == 1) + (p6 == 0 && p7 == 1) + (p7 == 0 && p8 == 1) + (p8 == 0 && p9 == 1) + (p9 == 0 && p2 == 1); int B = p2 + p3 + p4 + p5 + p6 + p7 + p8 + p9; int m1 = iter == 0 ? (p2 * p4 * p6) : (p2 * p4 * p8); int m2 = iter == 0 ? (p4 * p6 * p8) : (p2 * p6 * p8); if (A == 1 && (B >= 2 && B <= 6) && m1 == 0 && m2 == 0) marker.at<uchar>(i,j) = 1; } } im &= ~marker; } // Function for thinning any given binary image within the range of 0-255. If not you should first make sure that your image has this range preset and configured! void thinning(Mat& im) { // Enforce the range tob e in between 0 - 255 im /= 255; Mat prev = Mat::zeros(im.size(), CV_8UC1); Mat diff; do { thinningIteration(im, 0); thinningIteration(im, 1); absdiff(im, prev, diff); im.copyTo(prev); } while (countNonZero(diff) > 0); im *= 255; } The code above can then simply be applied to our previous steps by calling the thinning function on top of our previous binary generated image. The code for this is: Apply thinning algorithm Mat input_thinned = input_binary.clone(); thinning(input_thinned); This will result in the following output: Comparison between binarized and thinned fingerprint image using skeletization techniques. When we got this skeleton image, the following step would be to look for crossing points on the ridges of the fingerprint, which are being then called minutiae points. We can do this by a keypoint detector that looks at a large change in local contrast, like the Harris corner detector. Since the Harris corner detector is both able to detect strong corners and edges, this is ideally for the fingerprint problem, where the most important minutiae are short edges and bifurcation, the positions where edges come together. More information about minutae points and Harris Corner detection can be found in the following publications: Ross, Arun A., Jidnya Shah, and Anil K. Jain. "Toward reconstructing fingerprints from minutiae points." Defense and Security. International Society for Optics and Photonics, 2005. Harris, Chris, and Mike Stephens. "A combined corner and edge detector." Alvey vision conference. Vol. 15. 1988. Calling the Harris Corner operation on a skeletonized and binarized image in OpenCV is quite straightforward. The Harris corners are stored as positions corresponding in the image with their cornerness response value. If we want to detect points with a certain cornerness, than we should simply threshold the image. Mat harris_corners, harris_normalised; harris_corners = Mat::zeros(input_thinned.size(), CV_32FC1); cornerHarris(input_thinned, harris_corners, 2, 3, 0.04, BORDER_DEFAULT); normalize(harris_corners, harris_normalised, 0, 255, NORM_MINMAX, CV_32FC1, Mat()); We now have a map with all the available corner responses rescaled to the range of [0 255] and stored as float values. We can now manually define a threshold, that will generate a good amount of keypoints for our application. Playing around with this parameter could improve performance in other cases. This can be done by using the following code snippet: float threshold = 125.0; vector<KeyPoint> keypoints; Mat rescaled; convertScaleAbs(harris_normalised, rescaled); Mat harris_c(rescaled.rows, rescaled.cols, CV_8UC3); Mat in[] = { rescaled, rescaled, rescaled }; int from_to[] = { 0,0, 1,1, 2,2 }; mixChannels( in, 3, &harris_c, 1, from_to, 3 ); for(int x=0; x<harris_normalised.cols; x++){ for(int y=0; y<harris_normalised.rows; y++){ if ( (int)harris_normalised.at<float>(y, x) > threshold ){ // Draw or store the keypoint location here, just like you decide. In our case we will store the location of the keypoint circle(harris_c, Point(x, y), 5, Scalar(0,255,0), 1); circle(harris_c, Point(x, y), 1, Scalar(0,0,255), 1); keypoints.push_back( KeyPoint (x, y, 1) ); } } } Comparison between thinned fingerprint and Harris corner response, as well as the selected Harris corners. Now that we have a list of keypoints we will need to create some of formal descriptor of the local region around that keypoint to be able to uniquely identify it among other keypoints. Chapter 3, Recognizing facial expressions with machine learning, discusses in more detail the wide range of keypoints out there. In this article, we will mainly focus on the process. Feel free to adapt the interface with other keypoint detectors and descriptors out there, for better or for worse performance. Since we have an application where the orientation of the thumb can differ (since it is not a fixed position), we want a keypoint descriptor that is robust at handling these slight differences. One of the mostly used descriptors for that is the SIFT descriptor, which stands for scale invariant feature transform. However SIFT is not under a BSD license and can thus pose problems to use in commercial software. A good alternative in OpenCV is the ORB descriptor. In OpenCV you can implement it in the following way. Ptr<Feature2D> orb_descriptor = ORB::create(); Mat descriptors; orb_descriptor->compute(input_thinned, keypoints, descriptors); This will enable us to calculate only the descriptors using the ORB approach, since we already retrieved the location of the keypoints using the Harris corner approach. At this point we can retrieve a descriptor for each detected keypoint of any given fingerprint. The descriptors matrix will contain a row for each keypoint containing the representation. Let us now start from the case where we have only a single reference image for each fingerprint. In that case we will have a database containing a set of feature descriptors for the training persons in the database. We then have a single new entry, consisting of multiple descriptors for the keypoints found at registration time. We now have to match these descriptors to the descriptors stored in the database, to see which one has the best match. The most simple way is by performing a brute force matching using the hamming distance criteria between descriptors of different keypoints. // Imagine we have a vector of single entry descriptors as a database // We will still need to fill those once we compare everything, by using the code snippets above vector<Mat> database_descriptors; Mat current_descriptors; // Create the matcher interface Ptr<DescriptorMatcher> matcher = DescriptorMatcher::create("BruteForce-Hamming"); // Now loop over the database and start the matching vector< vector< DMatch > > all_matches; for(int entry=0; i<database_descriptors.size();entry++){ vector< DMatch > matches; matcheràmatch(database_descriptors[entry], current_descriptors, matches); all_matches.push_back(matches); } We now have all the matches stored as DMatch objects. This means that for each matching couple we will have the original keypoint, the matched keypoint and a floating point score between both matches, representing the distance between the matched points. The idea about finding the best match seems pretty straightforward. We take a look at the amount of matches that have been returned by the matching process and weigh them by their Euclidean distance in order to add some certainty. We then look for the matching process that yielded the biggest score. This will be our best match and the match we want to return as the selected one from the database. If you want to avoid an imposter getting assigned to the best matching score, you can again add a manual threshold on top of the scoring to avoid matches that are not good enough, to be ignored. However it is possible, and should be taken into consideration, that if you increase the score to high, that people with a little change will be rejected from the system, like for example in the case where someone cuts his finger and thus changing his pattern to drastically. Fingerprint matching process visualized. To summarize, we saw how to detect fingerprints and implement it using OpenCV 3. Resources for Article: Further resources on this subject: Making subtle color shifts with curves [article] Tracking Objects in Videos [article] Hand Gesture Recognition Using a Kinect Depth Sensor [article]
Read more
  • 0
  • 1
  • 26860
article-image-style-management-qgis
Packt
06 Jul 2015
11 min read
Save for later

Style Management in QGIS

Packt
06 Jul 2015
11 min read
In this article by Alexander Bruy and Daria Svidzinska, authors of the book QGIS By Example, you will learn how to work with styles, including saving and loading them, using different styles, and working with the Style Manager. (For more resources related to this topic, see here.) Working with styles In QGIS, a style is a way of cartographic visualization that takes into account a layer’s individual and thematic features. It encompasses basic characteristics of symbology, such as the color and presence of fill, outline parameters, the use of markers, scale-dependent rendering, layer transparency, and interactions with other layers. Style incorporates not only rendering appearance, but also other things, such as labeling settings. A well-chosen style greatly simplifies data perception and readability, so it is important to learn how to work with styles to be able to represent your data the best way. Styling is an important part of data preparation, and QGIS provides many handy features that make this process much more productive and easier. Let’s look at some of them! Saving and loading styles Creating good-looking styles can be a time-consuming task, but the good thing is that once developed styles don’t go to nowhere. You can save them for further use in other projects. When you have finished polishing your style, it is wise to save it. Usually, this is done from the Layer Properties dialog, which can be opened from the layer's context menu. There is a Style button at the bottom of this dialog. It provides access to almost all actions that can be performed with the layer's style, including saving, loading, making the style default, and so on. The style can be saved to the file on the disk (this works for any layer type), or stored in the database (possible only for database-based layers). To save style in the file, perform these steps: Open the Layer Properties dialog. Click on the Style button at the bottom of the Properties dialog and go to the Save Style submenu: Choose one of the available formats. A standard file dialog will open. Navigate to the desired location in your filesystem and save the style. Currently QGIS provides support for the following formats of saving styles: QGIS style file: The style is saved as a .qml file, which is a native QGIS format used to store symbology definition and other layer properties. Style Layer Descriptor (SLD) file: The style is exported to a .sld file. SLD format is widely used in web cartography, for example, by applications such as GeoServer. It is necessary to mention that currently, SLD support in QGIS is a bit limited. Also, you should remember that while you can save any style (or renderer type) in SLD, during import, you will get either a rule-based or a single-symbol renderer. If you work with the spatial database, you may want to save layer styles in the same database, together with layers. Such a feature is very useful in corporate environments, as it allows you to assign multiple styles for a single layer and easily keep the styles in sync. Saving styles in the database currently works only for PostgreSQL and SpatiaLite. To save style in the database, follow these steps: Open the Layer Properties dialog. Click on the Style button at the bottom of the Properties dialog and go to the Save Style submenu. Select the Save in database (format) item, where format can be spatialite or postgres, depending on the database type: The Save style in database dialog opens. Enter the style name and (optional) description in the corresponding fields, and click on the OK button to save the style: The saved style can be loaded and applied to the layer. To load a style from the file, use these steps: Open the Layer Properties dialog from the context menu. Click on the Style button at the bottom of the Properties dialog and select the Load Style item. Choose the style file to load. Loading a style from the database is a bit different: Open the Layer Properties dialog from the context menu. Click on the Style button at the bottom of the Properties dialog and go to Load Style | From database. The Load style from database dialog opens. Select the style you want to load and click on the Load Style button. With all of these options, we can easily save styles in the format that meets our requirements and tasks. Copy and paste styles Very often, you need to apply mostly the same style with really minor differences to multiple layers. There are several ways of doing this. First, you can save the style (as described in the previous section) in one of the supported formats, and then apply this saved style to another layer and edit it. But there is simpler way. Starting from QGIS 1.8, you can easily copy and paste styles between layers. To copy a style from one layer to another, perform these steps: In the QGIS layer tree, select the source layer from which you want to copy the style. Right-click to open the context menu. Go to Styles | Copy Style to copy the style of the source layer to the clipboard. Now, in the QGIS layer tree, select the target layer. Right-click to open its context menu. Go to Styles | Paster Style to paste the previously copied style from the clipboard and apply it to the target layer. It is important to note that QGIS allows you to copy, for example, a polygonal style and apply it to a point or line layer. This may lead to incorrect layer rendering, or the layer can even disappear from the map even though it still present in the layer tree. Instead of using the layer context menu to copy and paste styles, you can use the QGIS main menu. Both of these actions (Copy Style and Paste Style) can be found in the Layer menu. The copied style can be pasted in a text editor. Just copy the style using the context menu or QGIS main menu, open the text editor, and press Ctrl + V (or another shortcut used in your system to paste data from the clipboard) to paste the style. Now you can study it. Also, with this feature, you can apply the same style to multiple layers at once. Copy the style as previously described. Then select in the QGIS layer tree all the layers that you want to style (use the Ctrl key and/or the Shift key to select multiple layers). When all the desired layers are selected, go to Layer | Paste Style. Voilà! Now the style is applied to all selected layers. Using multiple styles per layer Sometimes, you may need to show the same data with different styles on the map. The most common and well-known solution to do this is to duplicate the layer in the QGIS layer tree and change the symbology. QGIS 2.8 allows us to achieve the same result in a simpler and more elegant way. Now we can define multiple styles for a single layer and easily switch between them when necessary. This functionality is available from the layer context menu and the layer Properties dialog. By default, all layers have only one style, called default. To create an additional style, use these steps: Select the layer in the layer tree. Right-click to open the context menu. Go to Styles | Add. The New style dialog opens. Enter the name of the new style and click on OK: A new style will be added and become active. It is worth mentioning that after adding a new style, the layer's appearance will remain the same, as the new style inherits all the properties of the active style. Adjust the symbology and other settings according to your needs. These changes will affect only the current style; previously created styles will remain unchanged. You can add as many styles as you want. All available styles are listed in the layer context menu, at the bottom of the Styles submenu. The current (or active) style is marked with a checkbox. To switch to another style, just select its name in the menu. If necessary, you can rename the active style (go to Styles | Rename Current) or remove it (go to Styles | Remove Current). Also, the current style can be saved and copied as previously described. Moreover, it is worth mentioning that multiple styles are supported by the QGIS server. The available layer styles are displayed via the GetCapabilities response, and the user can request them in, for example, the GetMap request. This handy feature also works in the Print Composer. Using Style manager Style manager provides extended capabilities for symbology management, allowing the user to save developed symbols; tag and merge them into thematic groups; and edit, delete, import, or export ready-to-use predefined symbology sets. If you created a symbol and want it to be available for further use and management, you should first save it to the Symbol library by following these steps: In the Style section of the layer Properties window, click on the Save button underneath the symbol preview window. In the Symbol name window, type a name for the new symbol and click on OK: After that, the symbol will appear and become available from the symbol presets in the right part of the window. It will also become available for the Style Manager. The Style Manger window can be opened by: Clicking on the Open library button after going to Properties | Style Going to Settings | Style Manager The window consists of three sections: In the left section, you can see a tree view of the available thematic symbology groups (which, by default, don't contain any user-specified groups) In the right part, there are symbols grouped on these tabs: Marker (for point symbols), Line (for line symbols), Fill (for polygon symbols), and Color ramp (for gradient symbols). If you double-click on any symbol on these tabs, the Symbol selector window will be opened, where you can change any available symbol properties (Symbol layer type, Size, Fill and Outline colors, and so on). Similarly, you can use the Edit button to change the appearance of the symbol. The bottom section of the window contains symbol management buttons—Add, Remove, Edit, and Share—for groups and their items. Let's create a thematic group called Hydrology. It will include symbology for hydrological objects, whether they are linear (river, canal, and so on) or polygonal (lake, water area, and so on). For this, perform the following steps: Highlight the groups item in the left section of the Style manager window and click on the very first + button. When the New group appears, type the Hydrology name. Now you need to add some symbols to the newly emerged group. There are two approaches to doing this: Right-click on any symbol (or several by holding down the Ctrl key) you want to add, and select from its contextual shortcut Apply Group | Hydrology. Alternatively, highlight the Hydrology group in the groups tree, and from the button below, select Group Symbols, as shown in this screenshot: As a result, checkboxes will appear beside the symbols, and you can toggle them to add the necessary symbol (or symbols) to the group. After you have clicked on the Close button, the symbols will be added to the group. Once the group is created, you can use it for quick access to the necessary symbology by going to Properties | Style | Symbols in group, as shown in the following screenshot: Note that you can combine the symbology for different symbology types within a single group (Marker, Line, Fill, and Color ramp), but when you upload symbols in this group for a specific layer, the symbols will be filtered according to the layer geometry type (for example, Fill for the polygon layer type). Another available option is to create a so-called Smart Group, where you can flexibly combine various conditions to merge symbols into meaningful groups. As an example, the following screenshot shows how we can create a wider Water group that includes symbols that are not only present in Hydrology already, but are also tagged as blue: Use the Share button to Export or Import selected symbols from external sources. Summary This article introduced the different aspects of style management in QGIS: saving and loading styles, copying and pasting styles, and using multiple styles per layer. Resources for Article: Further resources on this subject: How Vector Features are Displayed [article] Geocoding Address-based Data [article] Editing attributes [article]
Read more
  • 0
  • 0
  • 26738

article-image-using-handlebars-express
Packt
10 Aug 2015
17 min read
Save for later

Using Handlebars with Express

Packt
10 Aug 2015
17 min read
In this article written by Paul Wellens, author of the book Practical Web Development, we cover a brief description about the following topics: Templates Node.js Express 4 Templates Templates come in many shapes or forms. Traditionally, they are non-executable files with some pre-formatted text, used as the basis of a bazillion documents that can be generated with a computer program. I worked on a project where I had to write a program that would take a Microsoft Word template, containing parameters like $first, $name, $phone, and so on, and generate a specific Word document for every student in a school. Web templating does something very similar. It uses a templating processor that takes data from a source of information, typically a database and a template, a generic HTML file with some kind of parameters inside. The processor then merges the data and template to generate a bunch of static web pages or dynamically generates HTML on the fly. If you have been using PHP to create dynamic webpages, you will have been busy with web templating. Why? Because you have been inserting PHP code inside HTML files in between the <?php and ?> strings. Your templating processor was the Apache web server that has many additional roles. By the time your browser gets to see the result of your code, it is pure HTML. This makes this an example of server side templating. You could also use Ajax and PHP to transfer data in the JSON format and then have the browser process that data using JavaScript to create the HTML you need. Combine this with using templates and you will have client side templating. Node.js What Le Sacre du Printemps by Stravinsky did to the world of classical music, Node.js may have done to the world of web development. At its introduction, it shocked the world. By now, Node.js is considered by many as the coolest thing. Just like Le Sacre is a totally different kind of music—but by now every child who has seen Fantasia has heard it—Node.js is a different way of doing web development. Rather than writing an application and using a web server to soup up your code to a browser, the application and the web server are one and the same. This may sound scary, but you should not worry as there is an entire community that developed modules you can obtain using the npm tool. Before showing you an example, I need to point out an extremely important feature of Node.js: the language in which you will write your web server and application is JavaScript. So Node.js gives you server side JavaScript. Installing Node.js How to install Node.js will be different, depending on your OS, but the result is the same everywhere. It gives you two programs: Node and npm. npm The node packaging manager (npm)is the tool that you use to look for and install modules. Each time you write code that needs a module, you will have to add a line like this in: var module = require('module'); The module will have to be installed first, or the code will fail. This is how it is done: npm install module or npm -g install module The latter will attempt to install the module globally, the former, in the directory where the command is issued. It will typically install the module in a folder called node_modules. node The node program is the command to use to start your Node.js program, for example: node myprogram.js node will start and interpret your code. Type Ctrl-C to stop node. Now create a file myprogram.js containing the following text: var http = require('http'); http.createServer(function (req, res) { res.writeHead(200, {'Content-Type': 'text/plain'}); res.end('Hello Worldn'); }).listen(8080, 'localhost'); console.log('Server running at http://localhost:8080'); So, if you installed Node.js and the required http module, typing node myprogram.js in a terminal window, your console will start up a web server. And, when you type http://localhost:8080 in a browser, you will see the world famous two word program example on your screen. This is the equivalent of getting the It works! thing, after testing your Apache web server installation. As a matter of fact, if you go to http://localhost:8080/it/does/not/matterwhat, the same will appear. Not very useful maybe, but it is a web server. Serving up static content This does not work in a way we are used to. URLs typically point to a file (or a folder, in which case the server looks for an index.html file) , foo.html, or bar.php, and when present, it is served up to the client. So what if we want to do this with Node.js? We will need a module. Several exist to do the job. We will use node-static in our example. But first we need to install it: npm install node-static In our app, we will now create not only a web server, but a fileserver as well. It will serve all the files in the local directory public. It is good to have all the so called static content together in a separate folder. These are basically all the files that will be served up to and interpreted by the client. As we will now end up with a mix of client code and server code, it is a good practice to separate them. When you use the Express framework, you have an option to have Express create these things for you. So, here is a second, more complete, Node.js example, including all its static content. hello.js, our node.js app var http = require('http'); var static = require('node-static'); var fileServer = new static.Server('./public'); http.createServer(function (req, res) { fileServer.serve(req,res); }).listen(8080, 'localhost'); console.log('Server running at http://localhost:8080'); hello.html is stored in ./public. <!DOCTYPE html> <html> <head> <meta charset="UTF-8" /> <title>Hello world document</title> <link href="./styles/hello.css" rel="stylesheet"> </head> <body> <h1>Hello, World</h1> </body> </html> hello.css is stored in public/styles. body { background-color:#FFDEAD; } h1 { color:teal; margin-left:30px; } .bigbutton { height:40px; color: white; background-color:teal; margin-left:150px; margin-top:50px; padding:15 15 25 15; font-size:18px; } So, if we now visit http://localhost:8080/hello, we will see our, by now too familiar, Hello World message with some basic styling, proving that our file server also delivered the CSS file. You can easily take it one step further and add JavaScript and the jQuery library and put it in, for example, public/js/hello.js and public/js/jquery.js respectively. Too many notes With Node.js, you only install the modules that you need, so it does not include the kitchen sink by default! You will benefit from that for as far as performance goes. Back in California, I have been a proud product manager of a PC UNIX product, and one of our coolest value-add was a tool, called kconfig, that would allow people to customize what would be inside the UNIX kernel, so that it would only contain what was needed. This is what Node.js reminds me of. And it is written in C, as was UNIX. Deja vu. However, if we wanted our Node.js web server to do everything the Apache Web Server does, we would need a lot of modules. Our application code needs to be added to that as well. That means a lot of modules. Like the critics in the movie Amadeus said: Too many notes. Express 4 A good way to get the job done with fewer notes is by using the Express framework. On the expressjs.com website, it is called a minimal and flexible Node.js web application framework, providing a robust set of features for building web applications. This is a good way to describe what Express can do for you. It is minimal, so there is little overhead for the framework itself. It is flexible, so you can add just what you need. It gives a robust set of features, which means you do not have to create them yourselves, and they have been tested by an ever growing community. But we need to get back to templating, so all we are going to do here is explain how to get Express, and give one example. Installing Express As Express is also a node module, we install it as such. In your project directory for your application, type: npm install express You will notice that a folder called express has been created inside node_modules, and inside that one, there is another collection of node-modules. These are examples of what is called middleware. In the code example that follows, we assume app.js as the name of our JavaScript file, and app for the variable that you will use in that file for your instance of Express. This is for the sake of brevity. It would be better to use a string that matches your project name. We will now use Express to rewrite the hello.js example. All static resources in the public directory can remain untouched. The only change is in the node app itself: var express = require('express'); var path = require('path'); var app = express(); app.set('port', process.env.PORT || 3000); var options = { dotfiles: 'ignore', extensions: ['htm', 'html'], index: false }; app.use(express.static(path.join(__dirname, 'public') , options )); app.listen(app.get('port'), function () { console.log('Hello express started on http://localhost:' + app.get('port') + '; press Ctrl-C to terminate.' ); }); This code uses so called middleware (static) that is included with express. There is a lot more available from third parties. Well, compared to our node.js example, it is about the same number of lines. But it looks a lot cleaner and it does more for us. You no longer need to explicitly include the HTTP module and other such things. Templating and Express We need to get back to templating now. Imagine all the JavaScript ecosystem we just described. Yes, we could still put our client JavaScript code in between the <script> tags but what about the server JavaScript code? There is no such thing as <?javascript ?> ! Node.js and Express, support several templating languages that allow you to separate layout and content, and which have the template system do the work of fetching the content and injecting it into the HTML. The default templating processor for Express appears to be Jade, which uses its own, albeit more compact than HTML, language. Unfortunately, that would mean that you have to learn yet another syntax to produce something. We propose to use handlebars.js. There are two reasons why we have chosen handlebars.js: It uses <html> as the language It is available on both the client and server side Getting the handlebars module for Express Several Express modules exist for handlebars. We happen to like the one with the surprising name express-handlebars. So, we install it, as follows: npm install express-handlebars Layouts I almost called this section templating without templates as our first example will not use a parameter inside the templates. Most websites will consist of several pages, either static or dynamically generated ones. All these pages usually have common parts; a header and footer part, a navigation part or menu, and so on. This is the layout of our site. What distinguishes one page from another, usually, is some part in the body of the page where the home page has different information than the other pages. With express-handlebars, you can separate layout and content. We will start with a very simple example. Inside your project folder that contains public, create a folder, views, with a subdirectory layout. Inside the layouts subfolder, create a file called main.handlebars. This is your default layout. Building on top of the previous example, have it say: <!doctype html> <html> <head> <title>Handlebars demo</title> </head> <link href="./styles/hello.css" rel="stylesheet"> <body> {{{body}}} </body> </html> Notice the {{{body}}} part. This token will be replaced by HTML. Handlebars escapes HTML. If we want our HTML to stay intact, we use {{{ }}}, instead of {{ }}. Body is a reserved word for handlebars. Create, in the folder views, a file called hello.handlebars with the following content. This will be one (of many) example of the HTML {{{body}}}, which will be replaced by: <h1>Hello, World</h1> Let’s create a few more june.handlebars with: <h1>Hello, June Lake</h1> And bodie.handlebars containing: <h1>Hello, Bodie</h1> Our first handlebars example Now, create a file, handlehello.js, in the project folder. For convenience, we will keep the relevant code of the previous Express example: var express = require('express'); var path = require('path'); var app = express(); var exphbs = require(‘express-handlebars’); app.engine('handlebars', exphbs({defaultLayout: 'main'})); app.set('view engine', 'handlebars'); app.set('port', process.env.PORT || 3000); var options = { dotfiles: 'ignore', etag: false, extensions: ['htm', 'html'], index: false }; app.use(express.static(path.join(__dirname, 'public') , options  )); app.get('/', function(req, res) { res.render('hello');   // this is the important part }); app.get('/bodie', function(req, res) { res.render('bodie'); }); app.get('/june', function(req, res) { res.render('june'); }); app.listen(app.get('port'),  function () { console.log('Hello express started on http://localhost:' + app.get('port') + '; press Ctrl-C to terminate.' ); }); Everything that worked before still works, but if you type http://localhost:3000/, you will see a page with the layout from main.handlebars and {{{body}}} replaced by, you guessed it, the same Hello World with basic markup that looks the same as our hello.html example. Let’s look at the new code. First, of course, we need to add a require statement for our express-handlebars module, giving us an instance of express-handlebars. The next two lines specify what the view engine is for this app and what the extension is that is used for the templates and layouts. We pass one option to express-handlebars, defaultLayout, setting the default layout to be main. This way, we could have different versions of our app with different layouts, for example, one using Bootstrap and another using Foundation. The res.render calls determine which views need to be rendered, so if you type http:// localhost:3000/june, you will get Hello, June Lake, rather than Hello World. But this is not at all useful, as in this implementation, you still have a separate file for each Hello flavor. Let’s create a true template instead. Templates In the views folder, create a file, town.handlebars, with the following content: {{!-- Our first template with tokens --}} <h1>Hello, {{town}} </h1> Please note the comment line. This is the syntax for a handlebars comment. You could HTML comments as well, of course, but the advantage of using handlebars comments is that it will not show up in the final output. Next, add this to your JavaScript file: app.get('/lee', function(req, res) { res.render('town', { town: "Lee Vining"}); }); Now, we have a template that we can use over and over again with different context, in this example, a different town name. All you have to do is pass a different second argument to the res.render call, and {{town}} in the template, will be replaced by the value of town in the object. In general, what is passed as the second argument is referred to as the context. Helpers The token can also be replaced by the output of a function. After all, this is JavaScript. In the context of handlebars, we call those helpers. You can write your own, or use some of the cool built-in ones, such as #if and #each. #if/else Let us update town.handlebars as follows: {{#if town}} <h1>Hello, {{town}} </h1> {{else}} <h1>Hello, World </h1> {{/if}} This should be self explanatory. If the variable town has a value, use it, if not, then show the world. Note that what comes after #if can only be something that is either true of false, zero or not. The helper does not support a construct such as #if x < y. #each A very useful built-in helper is #each, which allows you to walk through an array of things and generate HTML accordingly. This is an example of the code that could be inside your app and the template you could use in your view folder: app.js code snippet var californiapeople = {    people: [ {“name":"Adams","first":"Ansel","profession":"photographer",    "born"       :"SanFrancisco"}, {“name":"Muir","first":"John","profession":"naturalist",    "born":"Scotland"}, {“name":"Schwarzenegger","first":"Arnold",    "profession":"governator","born":"Germany"}, {“name":"Wellens","first":"Paul","profession":"author",    "born":"Belgium"} ]   }; app.get('/californiapeople', function(req, res) { res.render('californiapeople', californiapeople); }); template (californiapeople.handlebars) <table class=“cooltable”> {{#each people}}    <tr><td>{{first}}</td><td>{{name}}</td>    <td>{{profession}}</td></tr> {{/each}} </table> Now we are well on our way to do some true templating. You can also write your own helpers, which is beyond the scope of an introductory article. However, before we leave you, there is one cool feature of handlebars you need to know about: partials. Partials In web development, where you dynamically generate HTML to be part of a web page, it is often the case that you repetitively need to do the same thing, albeit on a different page. There is a cool feature in express-handlebars that allows you to do that very same thing: partials. Partials are templates you can refer to inside a template, using a special syntax and drastically shortening your code that way. The partials are stored in a separate folder. By default, that would be views/partials, but you can even use subfolders. Let's redo the previous example but with a partial. So, our template is going to be extremely petite: {{!-- people.handlebars inside views  --}}    {{> peoplepartial }} Notice the > sign; this is what indicates a partial. Now, here is the familiar looking partial template: {{!-- peoplepartialhandlebars inside views/partials --}} <h1>Famous California people </h1> <table> {{#each people}} <tr><td>{{first}}</td><td>{{name}}</td> <td>{{profession}}</td></tr> {{/each}} </table> And, following is the JavaScript code that triggers it: app.get('/people', function(req, res) { res.render('people', californiapeople); }); So, we give it the same context but the view that is rendered is ridiculously simplistic, as there is a partial underneath that will take care of everything. Of course, these were all examples to demonstrate how handlebars and Express can work together really well, nothing more than that. Summary In this article, we talked about using templates in web development. Then, we zoomed in on using Node.js and Express, and introduced Handlebars.js. Handlebars.js is cool, as it lets you separate logic from layout and you can use it server-side (which is where we focused on), as well as client-side. Moreover, you will still be able to use HTML for your views and layouts, unlike with other templating processors. For those of you new to Node.js, I compared it to what Le Sacre du Printemps was to music. To all of you, I recommend the recording by the Los Angeles Philharmonic and Esa-Pekka Salonen. I had season tickets for this guy and went to his inaugural concert with Mahler’s third symphony. PHP had not been written yet, but this particular performance I had heard on the radio while on the road in California, and it was magnificent. Check it out. And, also check out Express and handlebars. Resources for Article: Let's Build with AngularJS and Bootstrap The Bootstrap grid system MODx Web Development: Creating Lists
Read more
  • 0
  • 2
  • 26687

article-image-how-to-develop-a-simple-to-do-list-app-tutorial
Sugandha Lahoti
26 Sep 2018
11 min read
Save for later

How to develop a Simple To-Do List App [Tutorial]

Sugandha Lahoti
26 Sep 2018
11 min read
In this article, we will build a simple to-do list app that allows a user to add and display tasks. In this process, we will learn the following: How to build a user interface in Android Studio Working with ListViews How to work with Dialogs This article is taken from the book Learning Kotlin by building Android Applications by Eunice Adutwumwaa Obugyei and Natarajan Raman. This book will teach you programming in Kotlin including data types, flow control, lambdas, object-oriented, and functional programming while building  Android Apps. Creating the project Let's start by creating a new project in Android Studio, with the name TodoList. Select Add No Activity on the Add an Activity to Mobile screen: When the project creation is complete, create a Kotlin Activity by selecting File | New | Kotlin Activity, as shown in the following screenshot: This will start a New Android Activity wizard. On the Add an Activity to Mobile screen, select Basic Activity, as shown in the following screenshot: Now, check Launcher Activity on the Customize the Activity screen, and click the Finish button: Building your UI In Android, the code for your user interface is written in XML. You can build your UI by doing either of the following: Using the Android Studio Layout Editor Writing the XML code by hand Let's go ahead and start designing our TodoList app. Using the Android Studio layout editor Android Studio provides a layout editor, which gives you the ability to build your layouts by dragging widgets into the visual editor. This will auto-generate the XML code for your UI. Open the content_main.xml file. Make sure the Design tab at the bottom of the screen is selected, as shown in the following screenshot: To add a component to your layout, you just drag the item from the Palette on the left side of the screen. To find a component, either scroll through the items on the Palette, or click on the Palette search icon and search for the item you need. If the Palette is not showing on your screen, select View | Tool Windows | Palette to display it. Go ahead and add a ListView to your view. When a view is selected, its attributes are displayed in the XML Attributes editor on the right side of the screen. The Attributes editor allows you to view and edit the attributes of the selected component. Go ahead and make the following changes: Set the ID as list_view Change both the layout_width and layout_height attributes to match_parent If the Attributes editor is not showing; select View | Tool Windows | Attributes to display it. Now, select Text at the bottom of the editor window to view the generated XML code. You'll notice that the XML code now has a ListView placed within the ConstraintLayout:  A layout always has a root element. In the preceding code, ConstraintLayout is the root element. Instead of using the layout editor, you could have written the previous code yourself. The choice between using the layout editor or writing the XML code is up to you. You can use the option that you're most comfortable with. We'll continue to make additions to the UI as we go along. Now, build and run your code. as shown in the following screenshot: As you can see, the app currently doesn't have much to it. Let's go ahead and add a little more flesh to it. Since we'll use the FloatingActionButton as the button the user uses to add a new item to their to-do list, we need to change its icon to one that makes its purpose quite clear. Open the activity_main.xml file: One of the attributes of the android.support.design.widget.FloatingActionButton is app:srcCompat. This is used to specify the icon for the FloatingActionButton. Change its value from @android:drawable/ic_dialog_email to @android:drawable/ic_input_add. Build and run again. The FloatingActionButton at the bottom now looks like an Add icon, as shown in the following screenshot: Adding functionality to the user interface At the moment, when the user clicks on the Add button, a ticker shows at the bottom of the screen. This is because of the piece of code in the onCreate() method that defines and sets an OnClickListener to the FloatingActionButton: fab.setOnClickListener { view -> Snackbar.make(view, "Replace with your own action", Snackbar.LENGTH_LONG) .setAction("Action", null).show() } This is not ideal for our to-do list app. Let's go ahead and create a new method in the MainActivity class that will handle the click event: fun showNewTaskUI() { } The method currently does nothing. We'll add code to show the appropriate UI soon. Now, replace the code within the setOnClickListener() call with a call to the new method: fab.setOnClickListener { showNewTaskUI() } Adding a new task For adding a new task, we'll show the user an AlertDialog with an editable field. Let's start by building the UI for the dialog. Right-click the res/layout directory and select New | Layout resource file, as shown in the following screenshot: On the New Resource File window, change the Root element to LinearLayout and set the File name as dialog_new_task. Click OK to create the layout,  as shown in the following screenshot: Open the dialog_new_task layout and add an EditText view to the LinearLayout. The XML code in the layout should now look like this: The inputType attribute is used to specify what kind of data the field can take. By specifying this attribute, the user is shown an appropriate keyboard. For example, if the inputType is set to number, the numbers keyboard is displayed: Now, let's go ahead and add a few string resources we'll need for the next section. Open the res/values/strings.xml file and add the following lines of code to the resources tag: <string name="add_new_task_dialog_title">Add New Task</string> <string name="save">Save</string> The add_new_task_dialog_title string will be used as the title of our dialog The save string will be used as the text of a button on the dialog The best way to use an AlertDialog is by encapsulating it in a DialogFragment. The DialogFragment takes away the burden of handling the dialog's life cycle events. It also makes it easy for you to reuse the dialog in other activities. Create a new Kotlin class with the name NewTaskDialogFragment, and replace the class definition with the following lines of code: class NewTaskDialogFragment: DialogFragment() { // 1 // 2 interface NewTaskDialogListener { fun onDialogPositiveClick(dialog: DialogFragment, task: String) fun onDialogNegativeClick(dialog: DialogFragment) } var newTaskDialogListener: NewTaskDialogListener? = null // 3 // 4 companion object { fun newInstance(title: Int): NewTaskDialogFragment { val newTaskDialogFragment = NewTaskDialogFragment() val args = Bundle() args.putInt("dialog_title", title) newTaskDialogFragment.arguments = args return newTaskDialogFragment } } override fun onCreateDialog(savedInstanceState: Bundle?): Dialog { // 5 val title = arguments.getInt("dialog_title") val builder = AlertDialog.Builder(activity) builder.setTitle(title) val dialogView = activity.layoutInflater.inflate(R.layout.dialog_new_task, null) val task = dialogView.findViewById<EditText>(R.id.task) builder.setView(dialogView) .setPositiveButton(R.string.save, { dialog, id -> newTaskDialogListener?.onDialogPositiveClick(this, task.text.toString); }) .setNegativeButton(android.R.string.cancel, { dialog, id -> newTaskDialogListener?.onDialogNegativeClick(this) }) return builder.create() } override fun onAttach(activity: Activity) { // 6 super.onAttach(activity) try { newTaskDialogListener = activity as NewTaskDialogListener } catch (e: ClassCastException) { throw ClassCastException(activity.toString() + " must implement NewTaskDialogListener") } } } Let's take a closer look at what this class does: The class extends the DialogFragment class. It declares an interface with the name NewTaskDialogListener, which declares two methods: onDialogPositiveClick(dialog: DialogFragment, task: String) onDialogNegativeClick(dialog: DialogFragment) It declares a variable of type NewTaskDialogListener. It defines a method, newInstance(), in a companion object. By doing this, the method can be accessed without having to create an instance of the NewTaskDialogFragment class. The newInstance() method does the following: It takes an Int parameter named title It creates an instance of the NewTaskDialogFragment and passes the title as part of its arguments It returns the new instance of the NewTaskDialogFragment It overrides the onCreateDialog() method. This method does the following: It attempts to retrieve the title argument passed It instantiates an AlertDialog builder and assigns the retrieved title as the dialog's title It uses the LayoutInflater of the DialogFragment instance's parent activity to inflate the layout we created Then, it sets the inflated view as the dialog's view Sets two buttons to the dialog: Save and Cancel When the Save button is clicked, the text in the EditText will be retrieved and passed to the newTaskDialogListener variable via the onDialogPositiveClick() method In the onAttach() method, we attempt to assign the Activity object passed to the newTaskDialogListener variable created earlier. For this to work, the Activity object should implement the NewTaskDialogListener interface. Now, open the MainActivity class. Change the class declaration to include implementation of the NewTaskDialogListener. Your class declaration should now look like this: class MainActivity : AppCompatActivity(), NewTaskDialogFragment.NewTaskDialogListener { And, add implementations of the methods declared in the NewTaskDialogListener by adding the following methods to the MainActivity class: override fun onDialogPositiveClick(dialog: DialogFragment, task:String) { } override fun onDialogNegativeClick(dialog: DialogFragment) { } In the showNewTaskUI() method, add the following lines of code: val newFragment = NewTaskDialogFragment.newInstance(R.string.add_new_task_dialog_title) newFragment.show(fragmentManager, "newtask") In the preceding lines of code, the newInstance() method in NewTaskDialogFragment is called to generate an instance of the NewTaskDialogFragment class. The show() method of the DialogFragment is then called to display the dialog. Build and run. Now, when you click the Add button, you should see a dialog on your screen,  as shown in the following screenshot: As you may have noticed, nothing happens when you click the SAVE button. In the  onDialogPositiveClick() method, add the line of code shown here: Snackbar.make(fab, "Task Added Successfully", Snackbar.LENGTH_LONG).setAction("Action", null).show() As we may remember, this line of code displays a ticker at the bottom of the screen. Now, when you click the SAVE button on the New Task dialog, a ticker shows at the bottom of the screen. We're currently not storing the task the user enters. Let's create a collection variable to store any task the user adds. In the MainActivity class, add a new variable of type ArrayList<String>, and instantiate it with an empty ArrayList: private var todoListItems = ArrayList<String>() In the onDialogPositiveClick() method, place the following lines of code at the beginning of the method definition: todoListItems.add(task) listAdapter?.notifyDataSetChanged() This will add the task variable passed to the todoListItems data, and call notifyDataSetChanged() on the listAdapter to update the ListView. Saving the data is great, but our ListView is still empty. Let's go ahead and rectify that. Displaying data in the ListView To make changes to a UI element in the XML layout, you need to use the findViewById() method to retrieve the instance of the element in the corresponding Activity of your layout. This is usually done in the onCreate() method of the Activity. Open MainActivity.kt, and declare a new ListView instance variable at the top of the class: private var listView: ListView? = null Next, instantiate the ListView variable with its corresponding element in the layout. Do this by adding the following line of code at the end of the onCreate() method: listView = findViewById(R.id.list_view) To display data in a ListView, you need to create an Adapter, and give it the data to display and information on how to display that data. Depending on how you want the data displayed in your ListView, you can either use one of the existing Android Adapters, or create your own. For now, we'll use one of the simplest Android Adapters, ArrayAdapter. The ArrayAdapter takes an array or list of items, a layout ID, and displays your data based on the layout passed to it. In the MainActivity class, add a new variable, of type ArrayAdapter: private var listAdapter: ArrayAdapter<String>? = null Add the method shown here to the class: private fun populateListView() { listAdapter = ArrayAdapter(this, android.R.layout.simple_list_item_1, todoListItems) listView?.adapter = listAdapter } In the preceding lines of code, we create a simple ArrayAdapter and assign it to the listView as its Adapter. Now, add a call to the previous method in the onCreate() method: populateListView() Build and run. Now, when you click the Add button, you'll see your entry show up on the ListView, as shown in the following screenshot: In this article, we built a simple TodoList app that allows a user to add new tasks, and edit or delete an already added task. In the process, we learned to use ListViews and Dialogs. Next, to learn about the different datastore options available and how to use them to make our app more usable, read our book, Learning Kotlin by building Android Applications. 6 common challenges faced by Android App developers. Google plans to let the AMP Project have an open governance model, soon! Entry level phones to taste the Go edition of the Android 9.0 Pie version
Read more
  • 0
  • 0
  • 26411
article-image-how-does-elasticsearch-work-tutorial
Savia Lobo
30 Jul 2018
12 min read
Save for later

How does Elasticsearch work? [Tutorial]

Savia Lobo
30 Jul 2018
12 min read
Elasticsearch is much more than just a search engine; it supports complex aggregations, geo filters, and the list goes on. Best of all, you can run all your queries at a speed you have never seen before.  Elasticsearch, like any other open source technology, is very rapidly evolving, but the core fundamentals that power Elasticsearch don't change. In this article, we will briefly discuss how Elasticsearch works internally and explain the basic query APIs.  All the data in Elasticsearch is internally stored in  Apache Lucene as an inverted index. Although data is stored in Apache Lucene, Elasticsearch is what makes it distributed and provides the easy-to-use APIs. This Elasticsearch tutorial is an excerpt taken from the book,'Learning Elasticsearch' written by Abhishek Andhavarapu. Inverted index in Elasticsearch Inverted index will help you understand the limitations and strengths of Elasticsearch compared with the traditional database systems out there. Inverted index at the core is how Elasticsearch is different from other NoSQL stores, such as MongoDB, Cassandra, and so on. We can compare an inverted index to an old library catalog card system. When you need some information/book in a library, you will use the card catalog, usually at the entrance of the library, to find the book. An inverted index is similar to the card catalog. Imagine that you were to build a system like Google to search for the web pages mentioning your search keywords. We have three web pages with Yoda quotes from Star Wars, and you are searching for all the documents with the word fear. Document1: Fear leads to anger Document2: Anger leads to hate Document3: Hate leads to suffering In a library, without a card catalog to find the book you need, you would have to go to every shelf row by row, look at each book title, and see whether it's the book you need. Computer-based information retrieval systems do the same. Without the inverted index, the application has to go through each web page and check whether the word exists in the web page. An inverted index is similar to the following table. It is like a map with the term as a key and list of the documents the term appears in as value. Term Document Fear 1 Anger 1,2 Hate 2,3 Suffering 3 Leads 1,2,3 Once we construct an index, as shown in this table, to find all the documents with the term fear is now just a lookup. Just like when a library gets a new book, the book is added to the card catalog, we keep building an inverted index as we encounter a new web page. The preceding inverted index takes care of simple use cases, such as searching for the single term. But in reality, we query for much more complicated things, and we don't use the exact words. Now let's say we encountered a document containing the following: Yosemite national park may be closed for the weekend due to forecast of substantial rainfall We want to visit Yosemite National Park, and we are looking for the weather forecast in the park. But when we query for it in the human language, we might query something like weather in yosemite or rain in yosemite. With the current approach, we will not be able to answer this query as there are no common terms between the query and the document, as shown: Document Query rainfall rain To be able to answer queries like this and to improve the search quality, we employ various techniques such as stemming, synonyms discussed in the following sections. Stemming Stemming is the process of reducing a derived word into its root word. For example, rain, raining, rained, rainfall has the common root word "rain". When a document is indexed, the root word is stored in the index instead of the actual word. Without stemming, we end up storing rain, raining, rained in the index, and search relevance would be very low. The query terms also go through the stemming process, and the root words are looked up in the index. Stemming increases the likelihood of the user finding what he is looking for. When we query for rain in yosemite, even though the document originally had rainfall, the inverted index will contain term rain. We can configure stemming in Elasticsearch using Analyzers. Synonyms Similar to rain and raining, weekend and sunday mean the same thing. The document might not contain Sunday, but if the information retrieval system can also search for synonyms, it will significantly improve the search quality. Human language deals with a lot of things, such as tense, gender, numbers. Stemming and synonyms will not only improve the search quality but also reduce the index size by removing the differences between similar words. More examples: Pen, Pen[s] -> Pen Eat, Eating  -> Eat Phrase search As a user, we almost always search for phrases rather than single words. The inverted index in the previous section would work great for individual terms but not for phrases. Continuing the previous example, if we want to query all the documents with a phrase anger leads to in the inverted index, the previous index would not be sufficient. The inverted index for terms anger and leads is shown below: Term Document Anger 1,2 Leads 1,2,3 From the preceding table, the words anger and leads exist both in document1 and document2. To support phrase search along with the document, we also need to record the position of the word in the document. The inverted index with word position is shown here: Term Document Fear 1:1 Anger 1:3, 2:1 Hate 2:3, 3:1 Suffering 3:3 Leads 1:2, 2:2, 3:2 Now, since we have the information regarding the position of the word, we can search if a document has the terms in the same order as the query. Term Document anger 1:3, 2:1 leads 1:2, 2:2 Since document2 has anger as the first word and leads as the second word, the same order as the query, document2 would be a better match than document1. With the inverted index, any query on the documents is just a simple lookup. This is just an introduction to inverted index; in real life, it's much more complicated, but the fundamentals remain the same. When the documents are indexed into Elasticsearch, documents are processed into the inverted index. Scalability and availability in Elasticsearch Let's say you want to index a billion documents; having just a single machine might be very challenging. Partitioning data across multiple machines allows Elasticsearch to scale beyond what a single machine do and support high throughput operations. Your data is split into small parts called shards. When you create an index, you need to tell Elasticsearch the number of shards you want for the index and Elasticsearch handles the rest for you. As you have more data, you can scale horizontally by adding more machines. We will go in to more details in the sections below. There are type of shards in Elasticsearch - primary and replica. The data you index is written to both primary and replica shards. Replica is the exact copy of the primary. In case of the node containing the primary shard goes down, the replica takes over. This process is completely transparent and managed by Elasticsearch. We will discuss this in detail in the Failure Handling section below. Since primary and replicas are the exact copies, a search query can be answered by either the primary or the replica shard. This significantly increases the number of simultaneous requests Elasticsearch can handle at any point in time. As the index is distributed across multiple shards, a query against an index is executed in parallel across all the shards. The results from each shard are then gathered and sent back to the client. Executing the query in parallel greatly improves the search performance. Now, we will discuss the relation between node, index and shard. Relation between node, index, and shard Shard is often the most confusing topic when I talk about Elasticsearch at conferences or to someone who has never worked on Elasticsearch. In this section, I want to focus on the relation between node, index, and shard. We will use a cluster with three nodes and create the same index with multiple shard configuration, and we will talk through the differences. Three shards with zero replicas We will start with an index called esintroduction with three shards and zero replicas. The distribution of the shards in a three node cluster is as follows: In the above screenshot, shards are represented by the green squares. We will talk about replicas towards the end of this discussion. Since we have three nodes(servers) and three shards, the shards are evenly distributed across all three nodes. Each node will contain one shard. As you index your documents into the esintroduction index, data is spread across the three shards. Six shards with zero replicas Now, let's recreate the same esintroduction index with six shards and zero replicas. Since we have three nodes (servers) and six shards, each node will now contain two shards. The esintroduction index is split between six shards across three nodes. The distribution of shards for an index with six shards is as follows: The esintroduction index is spread across three nodes, meaning these three nodes will handle the index/query requests for the index. If these three nodes are not able to keep up with the indexing/search load, we can scale the esintroduction index by adding more nodes. Since the index has six shards, you could add three more nodes, and Elasticsearch automatically rearranges the shards across all six nodes. Now, index/query requests for the esintroduction index will be handled by six nodes instead of three nodes. If this is not clear, do not worry, we will discuss more about this as we progress in the book. Six shards with one replica Let's now recreate the same esintroduction index with six shards and one replica, meaning the index will have 6 primary shards and 6 replica shards, a total of 12 shards. Since we have three nodes (servers) and twelve shards, each node will now contain four shards. The esintroduction index is split between six shards across three nodes. The green squares represent shards in the following figure. The solid border represents primary shards, and replicas are the dotted squares: As we discussed before, the index is distributed into multiple shards across multiple nodes. In a distributed environment, a node/server can go down due to various reasons, such as disk failure, network issue, and so on. To ensure availability, each shard, by default, is replicated to a node other than where the primary shard exists. If the node containing the primary shard goes down, the shard replica is promoted to primary, and the data is not lost, and you can continue to operate on the index. In the preceding figure, the esintroduction index has six shards split across the three nodes. The primary of shard 2 belongs to node elasticsearch 1, and the replica of the shard 2 belongs to node elasticsearch 3. In the case of the elasticsearch 1 node going down, the replica in elasticsearch 3 is promoted to primary. This switch is completely transparent and handled by Elasticsearch. Distributed search One of the reasons queries executed on Elasticsearch are so fast is because they are distributed. Multiple shards act as one index. A search query on an index is executed in parallel across all the shards. Let's take an example: in the following figure, we have a cluster with two nodes: Node1, Node2 and an index named chapter1 with two shards: S0, S1 with one replica: Assuming the chapter1 index has 100 documents, S1 would have 50 documents, and S0 would have 50 documents. And you want to query for all the documents that contain the word Elasticsearch. The query is executed on S0 and S1 in parallel. The results are gathered back from both the shards and sent back to the client. Imagine, you have to query across million of documents, using Elasticsearch the search can be distributed. For the application I'm currently working on, a query on more than 100 million documents comes back within 50 milliseconds; which is simply not possible if the search is not distributed. Failure handling in Elasticsearch Elasticsearch handles failures automatically. This section describes how the failures are handled internally. Let's say we have an index with two shards and one replica. In the following diagram, the shards represented in solid line are primary shards, and the shards in the dotted line are replicas: As shown in preceding diagram, we initially have a cluster with two nodes. Since the index has two shards and one replica, shards are distributed across the two nodes. To ensure availability, primary and replica shards never exist in the same node. If the node containing both primary and replica shards goes down, the data cannot be recovered. In the preceding diagram, you can see that the primary shard S0 belongs to Node 1 and the replica shard S0 to the Node 2. Next, just like we discussed in the Relation between Node, Index and Shard section, we will add two new nodes to the existing cluster, as shown here: The cluster now contains four nodes, and the shards are automatically allocated to the new nodes. Each node in the cluster will now contain either a primary or replica shard. Now, let's say Node2, which contains the primary shard S1, goes down as shown here: Since the node that holds the primary shard went down, the replica of S1, which lives in Node3, is promoted to primary. To ensure the replication factor of 1, a copy of the shard S1 is made on Node1. This process is known as rebalancing of the cluster. Depending on the application, the number of shards can be configured while creating the index. The process of rebalancing the shards to other nodes is entirely transparent to the user and handled automatically by Elasticsearch. We discussed inverted indexes, relation between nodes, index and shard, distributed search and how failures are handled automatically in Elasticsearch. Check out this book, 'Learning Elasticsearch' to know about handling document relationships, working with geospatial data, and much more. How to install Elasticsearch in Ubuntu and Windows Working with Kibana in Elasticsearch 5.x CRUD (Create Read, Update and Delete) Operations with Elasticsearch
Read more
  • 0
  • 2
  • 26331

article-image-installing-php-nuke
Packt
22 Feb 2010
10 min read
Save for later

Installing PHP-Nuke

Packt
22 Feb 2010
10 min read
The steps to install and configure PHP-Nuke are simple: Download and extract the PHP-Nuke files. Download and apply ChatServ's patches. Create the database for PHP-Nuke. Create a database user, and fill the database with data. Make some simple changes to the PHP-Nuke configuration file. Copy the PHP-Nuke files to the document root of the web server. Test it out! Let's get started. Downloading PHP-Nuke The latest version of PHP-Nuke can be downloaded at phpnuke.org downloads page: http://www.phpnuke.org/modules.php?name=Downloads&d_op=viewdownload&cid=1 You can also obtain older versions of PHP-Nuke, including version 1.0, from SourceForge: http://sourceforge.net/project/showfiles.php?group_id=7511&package_id=7622 SourceForge is the world's largest home of open-source projects. Many projects use SourceForge's facilities to host and maintain their projects. You can find almost anything you want on SourceForge—whether it is in a usable state or has been updated recently is another matter. Extracting PHP-Nuke Once you have downloaded PHP-Nuke, you should extract the contents of the PHP-Nuke ZIP archive to the root of your c: drive. You will have to create a folder called PHP-Nuke-7.8 in the root of your c: drive. (If you extract the files elsewhere, create the folder PHP-Nuke-7.8 and copy the contents of the main unzipped folder to this new folder). If you don't have a tool for extracting the files, you can download an evaluation edition (or buy a full edition) of WinZip from www.winzip.com. There are also free, powerful, extracting tools such as ZipGenius (http://www.zipgenius.it/index_eng.htm) and 7-Zip (http://sourceforge.net/projects/sevenzip/) among others. In the PHP-Nuke-7.8 folder, you will find three subfolders called html, sql, and upgrades. The upgrades folder contains scripts that handle upgrading the database between different versions, the sql folder contains the definition of the PHP-Nuke database that we will be working with, and the html folder contains the guts of your PHP-Nuke installation. The html folder contains all the PHP scripts, HTML files, images, CSS stylesheets, and so on that drive PHP-Nuke. Within the html folder are further subfolders; some of these include: modules: Contains the modules that make up your PHP-Nuke site. Modules are the essence of PHP-Nuke's operation; we look at them from article Your First Page onwards. blocks: Contains PHP-Nuke's blocks. Blocks are 'mini-functionality' units and usually provide snippet views of modules. We will look at blocks in article Managing the Site. language: Contains PHP-Nuke language files. These allow the language of PHP-Nuke's interface to be changed. images: Contains images used in the display of the PHP-Nuke site. themes: Contains the themes for PHP-Nuke. The use of themes allows you to completely change the look of a PHP-Nuke site with a click of a button. includes, db: Contain code to support the running of PHP-Nuke. The db folder, for example, contains database access code. admin: Contains code to power the administration area of your site. Downloading the Patches No software is without its flaws, and PHP-Nuke is no exception. After a release, the large user community invariably finds problems and potential security holes. Furthermore, PHP-Nuke also contains features such as its forum, which is in fact the phpBB application specially modified to work with PHP-Nuke. phpBB itself is updated on a regular basis to correct critical security vulnerabilities or to fix other problems, and consequently the corresponding part of PHP-Nuke also needs to be updated. Rather than releasing a new version of PHP-Nuke for these situations, patches for its various parts are released. ChatServ's patches from www.nukeresources.com are mostly concerned with variable validation, in other words, making sure that the variables used in the application are of the right type for storing in the database. This has been an area of weakness with many earlier versions of PHP-Nuke. The patches are often incorporated into subsequent versions of PHP-Nuke so that each new version becomes more robust. Note that you don't have to apply the patches, and PHP-Nuke will still work without them. However, by applying them you will have taken a good step towards improving the security of your site. If you navigate to http://www.nukeresources.com, there is a handy menu on the front page to access the patches: To obtain the patch corresponding to your version, click the link and you will be taken to the relevant file (of course, www.nukeresources is a PHP-Nuke powered site!). Click on the Nuke 7.8 link to go to the Downloads page of www.nukeresources.com. On this page, clicking the Download this file Now! link will download the patches for PHP-Nuke 7.8. The name of this file will be of the form 78patched.tar.gz. This is a GZIP compressed file that contains all the patches that we are about to apply. The GZIP file can be extracted with WinZip, or any of the other utilities we discussed earlier. The patches are simply modified versions of the original PHP-Nuke files. The original files have been modified to address various security issues that may have been identified since the initial release, or maybe since the last version of the patch. Applying the Patches To apply the patches, first we need to extract the 78patched.tar.gz file. We will extract the files into a folder called patches that we will create in the PHP-Nuke-7.8 folder. After extracting the files, copy the contents of the patches folder to your html folder. Do not copy the patches folder, copy its contents. The patches folder contains files that replace the files in the default installation, and you get a Confirm File Replace window. Select Yes for all the files, and when the copying is complete, your installation is ready to go. We have performed this patching immediately after installing PHP-Nuke, but we could have done this at any time. Doing it at this point is more sensible as it means that we are working on the most secure version of PHP-Nuke. Also, the patch process we have described here overwrites existing PHP-Nuke installation files. If you have modified these files, then the changes will be lost on applying the patch. Thus applying the patches later without disturbing any of your changes becomes more demanding. There is one further thing to watch for after applying the patches. You may find that the patched files have had their permissions set to read-only, and that you are unable to modify the files. To modify the files (and we do have to modify at least the config.php file in this article) you will need to remove this setting. You can do this on Windows by right-clicking on the file, selecting Properties from the menu, unchecking the Read-only setting, and clicking the OK button: We've done almost all the work with the files that we need to; now we turn our attention to creating and populating PHP-Nuke's database. Preparing the PHP-Nuke Database We'll be using the phpMyAdmin tool to do our database work. phpMyAdmin is part of the XAMPP installation (detailed in Appendix A), or can be downloaded from www.phpmyadmin.net, if you don't already have it. phpMyAdmin provides a powerful web interface for working with your MySQL databases. First of all, open your browser and navigate to http://localhost/phpmyadmin/, or whatever the location of your phpMyAdmin installation is: Creating the Database We need to create an empty database for PHP-Nuke to hold all the data about our site. To do this, we simply enter a name for our database into the Create new database textbox: We will call our database nuke. Enter this, and click the Create button. The name you give doesn't particularly matter, as long as it is not the name of some already existing database. If you try to use the same name as an already existing database, phpMyAdmin will inform you of this, and no action will be taken. The exact name isn't particularly important at this point because there is another configuration step coming up, which requires us to tell PHP-Nuke the name of the database we've created for it. After clicking Create, the screen will reload and you will be notified of the successful creation of your database: Creating a Database User Before we start populating the database, we will create a database user that can access only the PHP-Nuke database. This user is not a human, but will be used by PHP-Nuke to connect to the database while it performs its data-handling activities. The advantage of creating a database user is that it adds an extra level of security to our installation. PHP-Nuke will be able to work with data only in this database of the MySQL server, and no other. Also, PHP-Nuke will be restricted in the operations it can perform on the tables in the database. We will need to create a username for this boxed-in user to access the nuke database. Let's call our user nuker and go with the password nukepassword. However, in order to add an extra level of security we will introduce some digits into nukepassword, and some other slight twists, to strengthen it, and so use the word No0kPassv0rd as our database user password. To create the database user, click the SQL tab, and enter the following into the Run SQL query/queries on database textbox: GRANT ALL PRIVILEGES ON nuke.* TO nuker@localhost IDENTIFIED BY 'No0kPassv0rd' WITH GRANT OPTION Your screen should look like this: Click the Go button, and the database user will be created: Populating the Database Now we are ready to fill our database with data for PHP-Nuke. This doesn't mean we start typing the data in ourselves; the data comes with the PHP-Nuke installation. This data is found in a file called nuke.sql in the sql folder of the PHP-Nuke installation. This file contains a number of SQL statements that define the tables within the database and also fill them with 'raw' data for the site. However, before we fill the database with the tables from this file, we need to make a modification to this file. By default, the name of each database table in PHP-Nuke begins with nuke_. For example, there is a table with the name nuke_stories that holds information about stories, and a table called nuke_topics that holds information about story topics. These are just two of the tables; there are more than 90 in the standard installation. The word nuke_ is a 'table prefix', and is used to ensure that there are no clashes between the names of PHP-Nuke's tables and tables from another application in the same database, since the rest of the table name is descriptive of the data stored in the table, and other applications may have similarly named tables. What this does mean is that unless this table prefix is changed, the table names in your PHP-Nuke database will be known to anyone attempting to hack your site. Many of the typical attacks used to damage PHP-Nuke are based around the fact that the names of the tables in the database powering a PHP-Nuke site are known. By changing the table prefix to something less obvious, you have taken another step to making your site more secure.
Read more
  • 0
  • 6
  • 26064