Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
All Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

6719 Articles
article-image-improving-inspector-property-and-decorator-drawers
Packt
19 Aug 2015
10 min read
Save for later

Improving the Inspector with Property and Decorator Drawers

Packt
19 Aug 2015
10 min read
In this article by Angelo Tadres, author of the book Extending Unity with Editor Scripting, we will explore a way to create a custom GUI for our properties using Property Drawers. If you've worked on a Unity project for a long time, you know that the bigger your scripts get, the more unwieldy they become. All your public variables take up space in the Inspector Window, and as they accumulate, they begin to convert into one giant and scary monster. Sometimes, organization is the clue. So, in this article, you will learn how to improve your inspectors using Property and Decorator Drawers. (For more resources related to this topic, see here.) A Property Drawer is an attribute that allows you to control how the GUI of a Serializable class or property is displayed in the inspector window. An Attribute is a C# way of defining declarative tags, which you can place on certain entities in your source code to specify additional information. The information that attributes contain is retrieved at runtime through reflection. This approach significantly reduces the amount of work you have to do for the GUI customization because you don't need to write an entire Custom Inspector; instead, you can just apply appropriate attributes to variables in your scripts to tell the editor how you want those properties to be drawn. Unity has several Property Drawers implemented by default. Let's take a look at one of them called Range: using UnityEngine; public class RangeDrawerDemo : MonoBehaviour { [Range (0, 100)] public int IntValue = 50; } If you attach this script to a game object and then select it, you will see the following: Using the Range attribute, we rendered a slider that moves between 0 and 100 instead of the common int field. This is valuable in terms of validating the input for this field. Avoid possible mistakes such as using a negative value to define the radius of a sphere collider, and so on. Let's take a look to the rest of the built-in Property Drawers. Built-in Property Drawers The Unity documentation has information about the built-in property drawers, but there is no such place where you can check all the available ones listed. In this section, we will resolve this. Range The Range attribute restricts a float or int variable in a script to a specific range. When this attribute is used, the float or int will be shown as a slider in the inspector instead of the default number field: public RangeAttribute(float min, float max); public RangeAttribute(float min, float max); [Range (0, 1)] public float FloatRange = 0.5f; [Range (0, 100)] public int IntRange = 50; You will get the following output: Multiline The Multiline attribute is used to show a string value in a multiline text area. You can decide the number of lines of text to make room for. The default is 3 and the text doesn't wrap. The following is an example of the Multiline attribute: public MultilineAttribute(); public MultilineAttribute(int lines); [Multiline (2)] public string StringMultiline = "This text is using a multiline property drawer"; You will get the following output: TextArea The TextArea attribute allows a string to be edited with a height-flexible and scrollable text area. You can specify the minimum and maximum values; a scrollbar will appear if the text is bigger than the area available. Its behavior is better compared to Multiline. The following is an example of the TextArea attribute: public TextAreaAttribute(); public TextAreaAttribute(int minLines, int maxLines); [TextArea (2, 4)] public string StringTextArea = "This text is using a textarea property drawer"; You will get the following output: ContextMenu The ContextMenu attribute adds the method to the context menu of the component. When the user selects the context menu, the method will be executed. The method has to be nonstatic. In the following example, we call the method DoSomething, printing a log in the console: public ContextMenu(string name); [ContextMenu ("Do Something")] public void DoSomething() { Debug.Log ("DoSomething called..."); } You will get the following output: ContextMenuItem The ContextMenuItem attribute is used to add a context menu to a field that calls a named method. In the following example, we call a method to reset the value of the IntReset variable to 0: public ContextMenuItemAttribute(string name, string function); [ContextMenuItem("Reset this value", "Reset")] public int IntReset = 100; public void Reset() { IntReset = 0; } You will get the following output: Built-in Decorator Drawers There is another kind of drawer called Decorator Drawer. These drawers are similar in composition to the Property Drawers, but the main difference is that Decorator Drawers are designed to draw decoration in the inspector and are unassociated with a specific field. This means, while you can only declare one Property Drawer per variable, you can stack multiple decorator drawers in the same field. Let's take a look in the following built-in Decorator Drawers. Header This is the attribute that adds a header to some fields in the inspector: public HeaderAttribute(string header); [Header("This is a group of variables")] public int VarA = 10; public int VarB = 20; You will get the following output: Space The space attribute adds some spacing in the inspector: public SpaceAttribute(float height); public int VarC = 10; [Space(40)] public int VarD = 20; You will get the following output: Tooltip This Tooltip attribute specifies a tooltip for a field: public TooltipAttribute(string tooltip); [Tooltip("This is a tooltip")] public int VarE = 30; You will get the following output: Creating you own Property Drawers If you have a serializable parameter or structure that repeats constantly in your video game and you would like to improve how this renders in the Inspector, you can try to write your own Property Drawer. We will create a property drawer for an integer meant to be a variable to save time in seconds. This Property Drawer will draw a normal int field but also a label with the number of seconds converted to the m:s or h:m:s time format. To implement a Property Drawer, you must create two scripts: The attribute: This declares the attribute and makes it usable in your MonoBehaviour scripts. This will be part of your video game scripts. The drawer: This is responsible for rendering the custom GUI and handling the input of the user. This is placed inside a folder called Editor. The Editor folder is one of the several special folders Unity has. All scripts inside this folder will be treated as editor scripts rather than runtime scripts. For the first script, create one file inside your Unity project called TimeAttribute.cs and then add the following code: using UnityEngine; public class TimeAttribute : PropertyAttribute { public readonly bool DisplayHours; public TimeAttribute (bool displayHours = false) { DisplayHours = displayHours; } } Here we defined the name of the attribute and its parameters. You must create your attribute class extending from the PropertyAttribute class. The name of the class contains the suffix "attribute"; however, when you want to use the attribute over a certain property, the suffix is not needed. In this case, we will use Time and not TimeAttribute to use the property drawer. The TimeAttribute has an optional parameter called DisplayHours. The idea is to display a label under the int field with the time in m:s format by default; if the DisplayHours parameter is true, this will be displayed in h:m:s format. Now is the moment to implement the drawer. To do this, let's create a new script called TimeDrawer.cs inside an Editor folder: using UnityEngine; using UnityEditor; [CustomPropertyDrawer (typeof(TimeAttribute))] public class TimeDrawer : PropertyDrawer { public override float GetPropertyHeight (SerializedProperty property, GUIContent label) { return EditorGUI.GetPropertyHeight (property) * 2; } public override void OnGUI (Rect position, SerializedProperty property, GUIContent label) { if (property.propertyType == SerializedPropertyType.Integer) { property.intValue = EditorGUI.IntField (new Rect (position.x, position.y, position.width, position.height / 2), label, Mathf.Abs(property.intValue)); EditorGUI.LabelField (new Rect (position.x, position.y + position.height / 2, position.width, position.height / 2), " ", TimeFormat (property.intValue)); } else { EditorGUI.LabelField (position, label.text, "Use Time with an int."); } } private string TimeFormat (int seconds) { TimeAttribute time = attribute as TimeAttribute; if (time.DisplayHours) { return string.Format ("{0}:{1}:{2} (h:m:s)", seconds / (60 * 60), ((seconds % (60 * 60)) / 60).ToString ().PadLeft(2,'0'), (seconds % 60).ToString ().PadLeft(2,'0')); } else { return string.Format ("{0}:{1} (m:s)", (seconds / 60).ToString (), (seconds % 60).ToString ().PadLeft(2,'0')); } } } Property Drawers don't support layouts to create a GUI; for this reason, the class you must use here is EditorGUI instead of EditorGUILayout. Using this class requires a little extra effort; you need to define the Rect function that will contain the GUI element each time you want to use it. The CustomPropertyDrawer attribute is part of the UnityEditor namespace, and this is what Unity uses to bind a drawer with a Property attribute. In this case, we passed the TimeAttribute. Your must extend the TimeAttribute from the PropertyDrawer class, and in this way, you will have access to the core methods to create Property Drawers: GetPropertyHeight: This method is responsible for handling the height of the drawer. You need to overwrite this method in order to use it. In our case, we force the size of the drawer to be double. OnGUI: This is where you place all the code related to rendering the GUI. You can create Decorator Drawers too. You just need to follow the same steps we performed to create a Property Drawer, but instead of extending your Drawer from PropertyDrawer, you need to extend from DecoratorDrawer. Also, you will have access to the variable attribute. This has a reference to the attribute class we created, and with this we can access to their variables. To test our code, create a new script called TimeDrawerDemo.cs and add the following code: using UnityEngine; using UnityEditor; [CustomPropertyDrawer (typeof(TimeAttribute))] public class TimeDrawer : PropertyDrawer { public override float GetPropertyHeight (SerializedProperty property, GUIContent label) { return EditorGUI.GetPropertyHeight (property) * 2; } public override void OnGUI (Rect position, SerializedProperty property, GUIContent label) { if (property.propertyType == SerializedPropertyType.Integer) { property.intValue = EditorGUI.IntField (new Rect (position.x, position.y, position.width, position.height / 2), label, Mathf.Abs(property.intValue)); EditorGUI.LabelField (new Rect (position.x, position.y + position.height / 2, position.width, position.height / 2), " ", TimeFormat (property.intValue)); } else { EditorGUI.LabelField (position, label.text, "Use Time with an int."); } } private string TimeFormat (int seconds) { TimeAttribute time = attribute as TimeAttribute; if (time.DisplayHours) { return string.Format ("{0}:{1}:{2} (h:m:s)", seconds / (60 * 60), ((seconds % (60 * 60)) / 60).ToString ().PadLeft(2,'0'), (seconds % 60).ToString ().PadLeft(2,'0')); } else { return string.Format ("{0}:{1} (m:s)", (seconds / 60).ToString (), (seconds % 60).ToString ().PadLeft(2,'0')); } } } After compiling, attach this script to a game object. You will see this on the inspector: The time property uses the time attribute. We can check the three possible scenarios here: The attribute uses a property that is not an int The attribute has the variable DisplayHours = false The attribute has the variable DisplayHours = true A little change makes it easier to set up this data in our game objects. Summary In this article, we created a custom Property Drawer to be used in properties mean to store time in seconds, and in the process, you learned how these are implemented. We also explored the available built-in Property Drawers and Decorator Drawers in Unity. Applying this knowledge to your projects will enable you to add validation to sensible data in your properties and make your scripts more developer friendly. This will also allow you to have a professional look. Resources for Article: Further resources on this subject: Adding a Graphical User Interface [article] Animation features in Unity 5 [article] Event-driven Programming [article]
Read more
  • 0
  • 0
  • 7515

article-image-creating-controllers-blueprints
Packt
21 Sep 2015
8 min read
Save for later

Creating Controllers with Blueprints

Packt
21 Sep 2015
8 min read
In this article by Jack Stouffer, author of the book Mastering Flask, the more complex and powerful versions will be introduced, and we will turn our disparate view functions in cohesive wholes. We will also discuss the internals of how Flask handles the lifetime of an HTTP request and advanced ways to define Flask views. (For more resources related to this topic, see here.) Request setup, teardown, and application globals In some cases, a request-specific variable is needed across all view functions and needs to be accessed from the template as well. To achieve this, we can use Flask's decorator function @app.before_request and the object g. The function @app.before_request is executed every time before a new request is made. The Flask object g is a thread-safe store of any data that needs to be kept for each specific request. At the end of the request, the object is destroyed, and a new object is spawned at the start of a new request. For example, this code checks whether the Flask session variable contains an entry for a logged in user; if it exists, it adds the User object to g: from flask import g, session, abort, render_template @app.before_request def before_request(): if 'user_id' in session: g.user = User.query.get(session['user_id']) @app.route('/restricted') def admin(): if g.user is None: abort(403) return render_template('admin.html') Multiple functions can be decorated with @app.before_request, and they all will be executed before the requested view function is executed. There also exists a decorator @app.teardown_request, which is called after the end of every request. Keep in mind that this method of handling user logins is meant as an example and is not secure. Error pages Displaying browser's default error pages to the end user is jarring as the user loses all context of your app, and they must hit the back button to return to your site. To display your own templates when an error is returned with the Flask abort() function, use the errorhandler decorator function: @app.errorhandler(404) def page_not_found(error): return render_template('page_not_found.html'), 404 The errorhandler is also useful to translate internal server errors and HTTP 500 code into user friendly error pages. The app.errorhandler() function may take either one or many HTTP status code to define which code it will act on. The returning of a tuple instead of just an HTML string allows you to define the HTTP status code of the Response object. By default, this is set to 200. Class-based views In most Flask apps, views are handled by functions. However, when many views share common functionality or there are pieces of your code that could be broken out into separate functions, it would be useful to implement our views as classes to take advantage of inheritance. For example, if we have views that render a template, we could create a generic view class that keeps our code DRY: from flask.views import View class GenericView(View): def __init__(self, template): self.template = template super(GenericView, self).__init__() def dispatch_request(self): return render_template(self.template) app.add_url_rule( '/', view_func=GenericView.as_view( 'home', template='home.html' ) ) The first thing to note about this code is the dispatch_request() function in our view class. This is the function in our view that acts as the normal view function and returns an HTML string. The app.add_url_rule() function mimics the app.route() function as it ties a route to a function call. The first argument defines the route of the function, and the view_func parameter defines the function that handles the route. The View.as_view() method is passed to the view_func parameter because it transforms the View class into a view function. The first argument defines the name of the view function, so functions such as url_for() can route to it. The remaining parameters are passed to the __init__ function of the View class. Like the normal view functions, HTTP methods other than GET must be explicitly allowed for the View class. To allow other methods, a class variable containing the list of methods named methods must be added: class GenericView(View): methods = ['GET', 'POST'] … def dispatch_request(self): if request.method == 'GET': return render_template(self.template) elif request.method == 'POST': … Method class views Often, when functions handle multiple HTTP methods, the code can become difficult to read due to large sections of code nested within if statements: @app.route('/user', methods=['GET', 'POST', 'PUT', 'DELETE']) def users(): if request.method == 'GET': … elif request.method == 'POST': … elif request.method == 'PUT': … elif request.method == 'DELETE': … This can be solved with the MethodView class. MethodView allows each method to be handled by a different class method to separate concerns: from flask.views import MethodView class UserView(MethodView): def get(self): … def post(self): … def put(self): … def delete(self): … app.add_url_rule( '/user', view_func=UserView.as_view('user') ) Blueprints In Flask, a blueprint is a method of extending an existing Flask app. They provide a way of combining groups of views with common functionality and allow developers to break their app down into different components. In our architecture, the blueprints will act as our controllers. Views are registered to a blueprint; a separate template and static folder can be defined for it, and when it has all the desired content on it, it can be registered on the main Flask app to add blueprints' content. A blueprint acts much like a Flask app object, but is not actually a self-contained app. This is how Flask extensions provide views function. To get an idea of what blueprints are, here is a very simple example: from flask import Blueprint example = Blueprint( 'example', __name__, template_folder='templates/example', static_folder='static/example', url_prefix="/example" ) @example.route('/') def home(): return render_template('home.html') The blueprint takes two required parameters—the name of the blueprint and the name of the package—which are used internally in Flask, and passing __name__ to it will suffice. The other parameters are optional and define where the blueprint will look for files. Because templates_folder was specified, the blueprint will not look in the default template folder, and the route will render templates/example/home.html and not templates/home.html. The url_prefix option automatically adds the provided URI to the start of every route in the blueprint. So, the URL for the home view is actually /example/. The url_for() function will now have to be told which blueprint the requested route is in: {{ url_for('example.home') }} Also, the url_for() function will now have to be told whether the view is being rendered from within the same blueprint: {{ url_for('.home') }} The url_for() function will also look for static files in the specified static folder as well. To add the blueprint to our app: app.register_blueprint(example) Let's transform our current app to one that uses blueprints. We will first need to define our blueprint before all of our routes: blog_blueprint = Blueprint( 'blog', __name__, template_folder='templates/blog', url_prefix="/blog" ) Now, because the templates folder was defined, we need to move all of our templates into a subfolder of the templates folder named blog. Next, all of our routes need to have the @app.route function changed to @blog_blueprint.route, and any class view assignments now need to be registered to blog_blueprint. Remember that the url_for() function calls in the templates will also have to be changed to have a period prepended to then to indicate that the route is in the same blueprint. At the end of the file, right before the if __name__ == '__main__': statement, add the following: app.register_blueprint(blog_blueprint) Now all of our content is back on the app, which is registered under the blueprint. Because our base app no longer has any views, let's add a redirect on the base URL: @app.route('/') def index(): return redirect(url_for('blog.home')) Why blog and not blog_blueprint? Because blog is the name of the blueprint and the name is what Flask uses internally for routing. blog_blueprint is the name of the variable in the Python file. Summary We now have our app working inside a blueprint, but what does this give us? Let's say that we wanted to add a photo sharing function to our site, we would be able to group all the view functions into one blueprint with its own templates, static folder, and URL prefix without any fear of disrupting the functionality of the rest of the site. Resources for Article: Further resources on this subject: More about Julia [article] Optimization in Python [article] Symbolizers [article]
Read more
  • 0
  • 0
  • 7506

article-image-adding-postgis-layers-using-qgis-tutorial
Pravin Dhandre
26 Jul 2018
5 min read
Save for later

Adding PostGIS layers using QGIS [Tutorial]

Pravin Dhandre
26 Jul 2018
5 min read
Viewing tables as layers is great for creating maps or for simply working on a copy of the database outside the database. In this tutorial, we will establish a connection to our PostGIS database in order to add a table as a layer in QGIS (formerly known as Quantum GIS). Please navigate to the following site to install the latest version LTR of QGIS. On this page, click on Download Now and you will be able to choose a suitable operating system and the relevant settings. QGIS is available for Android, Linux, macOS X, and Windows. You might also be inclined to click on Discover QGIS to get an overview of basic information about the program along with features, screenshots, and case studies. This QGIS tutorial is an excerpt from a book written by Mayra Zurbaran,Pedro Wightman, Paolo Corti, Stephen Mather, Thomas Kraft and Bborie Park, titled PostGIS Cookbook - Second Edition. Loading Database... To begin, create the schema for this tutorial then, download data from the U.S. Census Bureau's FTP site: The shapefile is All Lines for Cuyahoga county in Ohio, which consist of roads and streams among other line features. Extract the ZIP file to your working directory and then load it into your database using shp2pgsql. Be sure to specify the spatial reference system, EPSG/SRID: 4269. When in doubt about using projections, use the service provided by the folks at OpenGeo at the following website: http://prj2epsg.org/search Use the following command to generate the SQL to load the shapefile: shp2pgsql -s 4269 -W LATIN1 -g the_geom -I tl_2012_39035_edges.shp chp11.tl_2012_39035_edges > tl_2012_39035_edges.sql How to do it... Now it's time to give the data we downloaded a look using QGIS. We must first create a connection to the database in order to access the table. Get connected and add the table as a layer by following the ensuing steps: Click on the Add PostGIS Layers icon: Click on the New button below the Connections drop-down menu. Create a new PostGIS connection. After the Add PostGIS Table(s) window opens, create a name for the connection and fill in a few parameters for your database, including Host, Port, Database, Username, and Password: Once you have entered all of the pertinent information for your database, click on the Test Connection button to verify that the connection is successful. If the connection is not successful, double-check for typos and errors. Additionally, make sure you are attempting to connect to a PostGIS-enabled database. If the connection is successful, go ahead and check the Save Username and Save Password checkboxes. This will prevent you from having to enter your login information multiple times throughout the exercise. Click on OK at the bottom of the menu to apply the connection settings. Now you can connect! Make sure the name of your PostGIS connection appears in the drop-down menu and then click on the Connect button. If you choose not to store your username and password, you will be asked to submit this information every time you try to access the database. Once connected, all schemas within the database will be shown and the tables will be made visible by expanding the target schema. Select the table(s) to be added as a layer by simply clicking on the table name or anywhere along its row. Selection(s) will be highlighted in blue. To deselect a table, click on it a second time and it will no longer be highlighted. Select the tl_2012_39035_edges table that was downloaded at the beginning of the tutorial and click on the Add button, as shown in the following screenshot: A subset of the table can also be added as a layer. This is accomplished by double-clicking on the desired table name. The Query Builder window will open, which aids in creating simple SQL WHERE clause statements. Add the roads by selecting the records where roadflg = Y. This can be done by typing a query or using the buttons within Query Builder: Click on the OK button followed by the Add button. A subset of the table is now loaded into QGIS as a layer. The layer is strictly a static, temporary copy of your database. You can make whatever changes you like to the layer and not affect the database table. The same holds true the other way around. Changes to the table in the database will have no effect on the layer in QGIS. If needed, you can save the temporary layer in a variety of formats, such as DXF, GeoJSON, KML, or SHP. Simply right-click on the layer name in the Layers panel and click on Save As. This will then create a file, which you can recall at a later time or share with others. The following screenshot shows the Cuyahoga county road network: You may also use the QGIS Browser Panel to navigate through the now connected PostGIS database and list the schemas and tables. This panel allows you to double-click to add spatial layers to the current project, providing a better user experience not only of connected databases, but on any directory of your machine: How it works... You have added a PostGIS layer into QGIS using the built-in Add PostGIS Table GUI. This was achieved by creating a new connection and entering your database parameters. Any number of database connections can be set up simultaneously. If working with multiple databases is more common for your workflows, saving all of the connections into one XML file and would save time and energy when returning to these projects in QGIS. To explore more on 3D capabilities of PostGIS, including LiDAR point clouds, read PostGIS Cookbook - Second Edition. Using R to implement Kriging – A Spatial Interpolation technique for Geostatistics data Top 7 libraries for geospatial analysis Learning R for Geospatial Analysis
Read more
  • 0
  • 0
  • 7498

article-image-ways-improve-performance-your-server-modsecurity-25
Packt
30 Nov 2009
13 min read
Save for later

Ways to improve performance of your server in ModSecurity 2.5

Packt
30 Nov 2009
13 min read
A typical HTTP request To get a better picture of the possible delay incurred when using a web application firewall, it helps to understand the anatomy of a typical HTTP request, and what processing time a typical web page download will incur. This will help us compare any added ModSecurity processing time to the overall time for the entire request. When a user visits a web page, his browser first connects to the server and downloads the main resource requested by the user (for example, an .html file). It then parses the downloaded file to discover any additional files, such as images or scripts, that it must download to be able to render the page. Therefore, from the point of view of a web browser, the following sequence of events happens for each file: Connect to web server. Request required file. Wait for server to start serving file. Download file. Each of these steps adds latency, or delay, to the request. A typical download time for a web page is on the order of hundreds of milliseconds per file for a home cable/DSL user. This can be slower or faster, depending on the speed of the connection and the geographical distance between the client and server. If ModSecurity adds any delay to the page request, it will be to the server processing time, or in other words the time from when the client has connected to the server to when the last byte of content has been sent out to the client. Another aspect that needs to be kept in mind is that ModSecurity will increase the memory usage of Apache. In what is probably the most common Apache configuration, known as "prefork", Apache starts one new child process for each active connection to the server. This means that the number of Apache instances increases and decreases depending on the number of client connections to the server.As the total memory usage of Apache depends on the number of child processes running and the memory usage of each child process, we should look at the way ModSecurity affects the memory usage of Apache. A real-world performance test In this section we will run a performance test on a real web server running Apache 2.2.8 on a Fedora Linux server (kernel 2.6.25). The server has an Intel Xeon 2.33 GHz dual-core processor and 2 GB of RAM. We will start out benchmarking the server when it is running just Apache without having ModSecurity enabled. We will then run our tests with ModSecurity enabled but without any rules loaded. Finally, we will test ModSecurity with a ruleset loaded so that we can draw conclusions about how the performance is affected. The rules we will be using come supplied with ModSecurity and are called the "core ruleset". The core ruleset The ModSecurity core ruleset contains over 120 rules and is shipped with the default ModSecurity source distribution (it's contained in the rules sub-directory). This ruleset is designed to provide "out of the box" protection against some of the most common web attacks used today. Here are some of the things that the core ruleset protects against: Suspicious HTTP requests (for example, missing User-Agent or Accept headers) SQL injection Cross-Site Scripting (XSS) Remote code injection File disclosure We will examine these methods of attack, but for now, let's use the core ruleset and examine how enabling it impacts the performance of your web service. Installing the core ruleset To install the core ruleset, create a new sub-directory named modsec under your Apache conf directory (the location will vary depending on your distribution). Then copy all the .conf files from the rules sub-directory of the source distribution to the new modsec directory: mkdir /etc/httpd/conf/modseccp/home/download/modsecurity-apache/rules/modsecurity_crs_*.conf /etc/httpd/conf/modsec Finally, enter the following line in your httpd.conf file and restart Apache to make it read the new rule files: # Enable ModSecurity core rulesetInclude conf/modsecurity/*.conf Putting the core rules in a separate directory makes it easy to disable them—all you have to do is comment out the above Include line in httpd.conf, restart Apache, and the rules will be disabled. Making sure it works The core ruleset contains a file named modsecurity_crs_10_config.conf. This file contains some of the basic configuration directives needed to turn on the rule engine and configure request and response body access. Since we have already configured these directives, we do not want this file to conflict with our existing configuration, and so we need to disable this. To do this, we simply need to rename the file so that it has a different extension as Apache only loads *.conf files with the Include directive we used above: $ mv modsecurity_crs_10_config.conf modsecurity_crs_10_config.conf.disabled Once we have restarted Apache, we can test that the core ruleset is loaded by attempting to access an URL that it should block. For example, try surfing to http://yourserver/ftp.exe and you should get the error message Method Not Implemented, ensuring that the core rules are loaded. Performance testing basics So what effect does loading the core ruleset have on web application response time and how do we measure this? We could measure the response time for a single request with and without the core ruleset loaded, but this wouldn't have any statistical significance—it could happen that just as one of the requests was being processed, the server started to execute a processor-intensive scheduled task, causing a delayed response time. The best way to compare the response times is to issue a large number of requests and look at the average time it takes for the server to respond. An excellent tool—and the one we are going to use to benchmark the server in the following tests—is called httperf. Written by David Mosberger of Hewlett Packard Research Labs, httperf allows you to simulate high workloads against a web server and obtain statistical data on the performance of the server. You can obtain the program at http://www.hpl.hp.com/research/linux/httperf/ where you'll also find a useful manual page in the PDF file format and a link to the research paper published together with the first version of the tool. Using httperf We'll run httperf with the options --hog (use as many TCP ports as needed), --uri/index.html (request the static web page index.html) and we'll use --num-conn 1000 (initiate a total of 1000 connections). We will be varying the number of requests per second (specified using --rate) to see how the server responds under different workloads. This is what the typical output from httperf looks like when run with the above options: $ ./httperf --hog --server=bytelayer.com --uri /index.html --num-conn1000 --rate 50Total: connections 1000 requests 1000 replies 1000 test-duration20.386 sConnection rate: 49.1 conn/s (20.4 ms/conn, <=30 concurrentconnections)Connection time [ms]: min 404.1 avg 408.2 max 591.3 median 404.5stddev 16.9Connection time [ms]: connect 102.3Connection length [replies/conn]: 1.000Request rate: 49.1 req/s (20.4 ms/req)Request size [B]: 95.0Reply rate [replies/s]: min 46.0 avg 49.0 max 50.0 stddev 2.0 (4samples)Reply time [ms]: response 103.1 transfer 202.9Reply size [B]: header 244.0 content 19531.0 footer 0.0 (total19775.0)Reply status: 1xx=0 2xx=1000 3xx=0 4xx=0 5xx=0CPU time [s]: user 2.37 system 17.14 (user 11.6% system 84.1% total95.7%)Net I/O: 951.9 KB/s (7.8*10^6 bps)Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0 The output shows us the number of TCP connections httperf initiated per second ("Connection rate"), the rate at which it requested files from the server ("Request rate"), and the actual reply rate that the server was able to provide ("Reply rate"). We also get statistics on the reply time—the "reply time – response" is the time taken from when the first byte of the request was sent to the server to when the first byte of the reply was received—in this case around 103 milliseconds. The transfer time is the time to receive the entire response from the server. The page we will be requesting in this case, index.html, is 20 KB in size which is a pretty average size for an HTML document. httperf requests the page one time per connection and doesn't follow any links in the page to download additional embedded content or script files, so the number of such links in the page is of no relevance to our test. Getting a baseline: Testing without ModSecurity When running benchmarking tests like this one, it's always important to get a baseline result so that you know the performance of your server when the component you're measuring is not involved. In our case, we will run the tests against the server when ModSecurity is disabled. This will allow us to tell which impact, if any, running with ModSecurity enabled has on the server. Response time The following chart shows the response time, in milliseconds, of the server when it is running without ModSecurity. The number of requests per second is on the horizontal axis: As we can see, the server consistently delivers response times of around 300 milliseconds until we reach about 75 requests per second. Above this, the response time starts increasing, and at around 500 requests per second the response time is almost a second per request. This data is what we will use for comparison purposes when looking at the response time of the server after we enable ModSecurity. Memory usage Finding the memory usage on a Linux system can be quite tricky. Simply running the Linux top utility and looking at the amount of free memory doesn't quite cut it, and the reason is that Linux tries to use almost all free memory as a disk cache. So even on a system with several gigabytes of memory and no memory-hungry processes, you might see a free memory count of only 50 MB or so. Another problem is that Apache uses many child processes, and to accurately measure the memory usage of Apache we need to sum the memory usage of each child process. What we need is a way to measure the memory usage of all the Apache child processes so that we can see how much memory the web server truly uses. To solve this, here is a small shell script that I have written that runs the ps command to find all the Apache processes. It then passes the PID of each Apache process to pmap to find the memory usage, and finally uses awk to extract the memory usage (in KB) for summation. The result is that the memory usage of Apache is printed to the terminal. The actual shell command is only one long line, but I've put it into a file called apache_mem.sh to make it easier to use: #!/bin/sh# apache_mem.sh# Calculate the Apache memory usageps -ef | grep httpd | grep ^apache | awk '{ print $2 }' | xargs pmap -x | grep 'total kB' | awk '{ print $3 }' | awk '{ sum += $1 } END { print sum }' Now, let's use this script to look at the memory usage of all of the Apache processes while we are running our performance test. The following graph shows the memory usage of Apache as the number of requests per second increases: Apache starts out consuming about 300 MB of memory. Memory usage grows steadily and at about 150 requests per second it starts climbing more rapidly. At 500 requests per second, the memory usage is over 2.4 GB—more than the amount of physical RAM of the server. The fact that this is possible is because of the virtual memory architecture that Linux (and all modern operating systems) use. When there is no more physical RAM available, the kernel starts swapping memory pages out to disk, which allows it to continue operating. However, since reading and writing to a hard drive is much slower than to memory, this starts slowing down the server significantly, as evidenced by the increase in response time seen in the previous graph. CPU usage In both of the tests above, the server's CPU usage was consistently around 1 to 2%, no matter what the request rate was. You might have expected a graph of CPU usage in the previous and subsequent tests, but while I measured the CPU usage in each test, it turned out to run at this low utilization rate for all tests, so a graph would not be very useful. Suffice it to say that in these tests, CPU usage was not a factor. ModSecurity without any loaded rules Now, let's enable ModSecurity—but without loading any rules—and see what happens to the response time and memory usage. Both SecRequestBodyAccess and SecResponseBodyAccess were set to On, so if there is any performance penalty associated with buffering requests and responses, we should see this now that we are running ModSecurity without any rules. The following graph shows the response time of Apache with ModSecurity enabled: We can see that the response time graph looks very similar to the response time graph we got when ModSecurity was disabled. The response time starts increasing at around 75 requests per second, and once we pass 350 requests per second, things really start going downhill. The memory usage graph is also almost identical to the previous one: Apache uses around 1.3 MB extra per child process when ModSecurity is loaded, which equals a total increase of memory usage of 26 MB for this particular setup. Compared to the total amount of memory Apache uses when the server is idle (around 300 MB) this equals an increase of about 10%. Mod Security with the core ruleset loaded Now for the really interesting test we'll run httperf against ModSecurity with the core ruleset loaded and look at what that does to the response time and memory usage. Response time The following graph shows the server response time with the core ruleset loaded: At first, the response time is around 340 ms, which is about 35 ms slower than in previous tests. Once the request rate gets above 50, the server response time starts deteriorating. As the request rates grows, the response time gets worse and worse, reaching a full 5 seconds at 100 requests per second. I have capped the graph at 100 requests per second, as the server performance has already deteriorated enough at this point to allow us to see the trend. We see that the point at which memory usage starts increasing has gone down from 75 to 50 requests per second now that we have enabled the core ruleset. This equals a reduction in the maximum number of requests per second the server can handle of 33%.
Read more
  • 0
  • 0
  • 7489

article-image-how-create-new-jsf-project
Packt
20 Jun 2011
17 min read
Save for later

How to Create a New JSF Project

Packt
20 Jun 2011
17 min read
  Java EE 6 Development with NetBeans 7 Develop professional enterprise Java EE applications quickly and easily with this popular IDE       Introduction to JavaServer faces Before JSF existed, most Java web applications were typically developed using non-standard web application frameworks such as Apache Struts, Tapestry, Spring Web MVC, or many others. These frameworks are built on top of the Servlet and JSP standards, and automate a lot of functionality that needs to be manually coded when using these APIs directly. Having a wide variety of web application frameworks available, often resulted in "analysis paralysis", that is, developers often spend an inordinate amount of time evaluating frameworks for their applications. The introduction of JSF to the Java EE specification resulted in having a standard web application framework available in any Java EE compliant application server. We don't mean to imply that other web application frameworks are obsolete or that they shouldn't be used at all. However, a lot of organizations consider JSF the "safe" choice since it is part of the standard and should be well supported for the foreseeable future. Additionally, NetBeans offers excellent JSF support, making JSF a very attractive choice. Strictly speaking, JSF is not a web application framework per se, but a component framework. In theory, JSF can be used to write applications that are not web-based, however, in practice JSF is almost always used for this purpose. In addition to being the standard Java EE component framework, one benefit of JSF is that it provides good support for tools vendors, allowing tools such as NetBeans to take advantage of the JSF component model with drag and drop support for components.   Developing our first JSF application From an application developer's point of view, a JSF application consists of a series of XHTML pages containing custom JSF tags, one or more JSF managed beans, and an optional configuration file named faces-config.xml. faces-config.xml used to be required in JSF 1.x, however, in JSF 2.0, some conventions were introduced that reduce the need for configuration. Additonally, a lot of JSF configuration can be specified using annotations, reducing, and in some cases, eliminating the need for this XML configuration file. Creating a new JSF project To create a new JSF project, we need to go to File | New Project, select the Java Web project category, and Web Application as the project type. After clicking Next>, we need to enter a project name, and optionally change other information for our project, although NetBeans provides sensible defaults. On the next page in the wizard, we can select the server, Java EE version, and context path of our application. In our example we will simply pick the default values. On the next page of the new project wizard, we can select what frameworks our web application will use. Unsurprisingly, for JSF applications we need to select the JavaServer Faces framework. When clicking on Finish, the wizard generates a skeleton JSF project for us, consisting of a single facelet file called index.xhtml, a web.xml configuration file. web.xml is the standard, optional configuration file needed for Java web applications, this file became optional in version 3.0 of the Servlet API, which was introduced with Java EE 6. In many cases, web.xml is not needed anymore, since most of the configuration options can now be specified via annotations. For JSF applications, however, it is a good idea to add one, since it allows us to specify the JSF project stage. <?xml version="1.0" encoding="UTF-8"?> <web-app version="3.0" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"> <context-param> <param-name>javax.faces.PROJECT_STAGE</param-name> <param-value>Development</param-value> </context-param> <servlet> <servlet-name>Faces Servlet</servlet-name> <servlet-class>javax.faces.webapp.FacesServlet</servlet-class> <load-on-startup>1</load-on-startup> </servlet> <servlet-mapping> <servlet-name>Faces Servlet</servlet-name> <url-pattern>/faces/*</url-pattern> </servlet-mapping> <session-config> <session-timeout> 30 </session-timeout> </session-config> <welcome-file-list> <welcome-file>faces/index.xhtml</welcome-file> </welcome-file-list> As we can see, NetBeans automatically sets the JSF project stage to Development, setting the project stage to development configures JSF to provide additional debugging help not present in other stages. For example, one common problem when developing a page is that while a page is being developed, validation for one or more of the fields on the page fails, but the developer has not added an <h:message> or <h:messages> tag to the page. When this happens and the form is submitted, the page seems to do nothing, or page navigation doesn't seem to be working. When setting the project stage to Development, these validation errors will automatically be added to the page, without the developer having to explicitly add one of these tags to the page (we should, of course, add the tags before releasing our code to production, since our users will not see the automatically generated validation errors). The following are the valid values for the javax.faces.PROJECT_STAGE context parameter for the faces servlet: Development Production SystemTest UnitTest The Development project stage adds additional debugging information to ease development. The Production project stage focuses on performance. The other two valid values for the project stage (SystemTest and UnitTest), allow us to implement our own custom behavior for these two phases. The javax.faces.application.Application class has a getProjectStage() method that allows us to obtain the current project stage. Based on the value of this method, we can implement the code that will only be executed in the appropriate stage. The following code snippet illustrates this: public void someMethod() { FacesContext facesContext = FacesContext.getCurrentInstance(); Application application = facesContext.getApplication(); ProjectStage projectStage = application.getProjectStage(); if (projectStage.equals(ProjectStage.Development)) { //do development stuff } else if (projectStage.equals(ProjectStage.Production)) { //do production stuff } else if (projectStage.equals(ProjectStage.SystemTest)) { // do system test stuff } else if (projectStage.equals(ProjectStage.UnitTest)) { //do unit test stuff } } As illustrated in the snippet above, we can implement the code to be executed in any valid project stage, based on the return value of the getProjectStage() method of the Application class. When creating a Java Web project using JSF, a facelet is automatically generated. The generated facelet file looks like this: <?xml version='1.0' encoding='UTF-8' ?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html > <h:head> <title>Facelet Title</title> </h:head> <h:body> Hello from Facelets </h:body> </html> As we can see, a facelet is nothing but an XHTML file using some facelets-specific XML name spaces. In the automatically generated page above, the following namespace definition allows us to use the "h" (for HTML) JSF component library: The above namespace declaration allows us to use JSF specific tags such as <h:head> and <h:body> which are a drop in replacement for the standard HTML/XHTML <head> and <body> tags, respectively. The application generated by the new project wizard is a simple, but complete JSF web application. We can see it in action by right-clicking on our project in the project window and selecting Run. At this point the application server is started (if it wasn't already running), the application is deployed and the default system browser opens, displaying our application's default page. Modifying our page to capture user data The generated application, of course, is nothing but a starting point for us to create a new application. We will now modify the generated index.xhtml file to collect some data from the user. The first thing we need to do is add an <h:form> tag to our page. The <h:form> tag is equivalent to the <form> tag in standard HTML pages. After typing the first few characters of the <h:form> tag into the page, and hitting Ctrl+Space, we can take advantage of NetBeans' excellent code completion. After adding the <h:form> tag and a number of additional JSF tags, our page now looks like this: <?xml version='1.0' encoding='UTF-8' ?> <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html > <h:head> <title>Registration</title> <h:outputStylesheet library="css" name="styles.css"/> </h:head> <h:body> <h3>Registration Page</h3> <h:form> <h:panelGrid columns="3" columnClasses="rightalign,leftalign,leftalign"> <h:outputLabel value="Salutation: " for="salutation"/> <h:selectOneMenu id="salutation" label="Salutation" value="#{registrationBean.salutation}" > <f:selectItem itemLabel="" itemValue=""/> <f:selectItem itemLabel="Mr." itemValue="MR"/> <f:selectItem itemLabel="Mrs." itemValue="MRS"/> <f:selectItem itemLabel="Miss" itemValue="MISS"/> <f:selectItem itemLabel="Ms" itemValue="MS"/> <f:selectItem itemLabel="Dr." itemValue="DR"/> </h:selectOneMenu> <h:message for="salutation"/> <h:outputLabel value="First Name:" for="firstName"/> <h:inputText id="firstName" label="First Name" required="true" value="#{registrationBean.firstName}" /> <h:message for="firstName" /> <h:outputLabel value="Last Name:" for="lastName"/> <h:inputText id="lastName" label="Last Name" required="true" value="#{registrationBean.lastName}" /> <h:message for="lastName" /> <h:outputLabel for="age" value="Age:"/> <h:inputText id="age" label="Age" size="2" value="#{registrationBean.age}"/> <h:message for="age"/> <h:outputLabel value="Email Address:" for="email"/> <h:inputText id="email" label="Email Address" required="true" value="#{registrationBean.email}"> </h:inputText> <h:message for="email" /> <h:panelGroup/> <h:commandButton id="register" value="Register" action="confirmation" /> </h:panelGrid> </h:form> </h:body> </html> The following screenshot illustrates how our page will be rendered at runtime: All JSF input fields must be inside an <h:form> tag. The <h:panelGrid> helps us to easily lay out JSF tags on our page. It can be thought of as a grid where other JSF tags will be placed. The columns attribute of the <h:panelGrid> tag indicates how many columns the grid will have, each JSF component inside the <h:panelGrid> component will be placed in an individual cell of the grid. When the number of components matching the value of the columns attribute (three in our example) has been placed inside <h:panelGrid>, a new row is automatically started. The following table illustrates how tags will be laid out inside an <h:panelGrid> tag: Each row in our <h:panelGrid> consists of an <h:outputLabel> tag, an input field, and an <h:message> tag. The columnClasses attribute of <h:panelGrid> allows us to assign CSS styles to each column inside the panel grid, its value attribute must consist of a comma separated list of CSS styles (defined in a CSS stylesheet). The first style will be applied to the first column, the second style will be applied to the second column, the third style will be applied to the third column, so on and so forth. Had our panel grid had more than three columns, then the fourth column would have been styled using the first style in the columnClasses attribute, the fifth column would have been styled using the second style in the columnClasses attribute, so on and so forth. If we wish to style rows in an <h:panelGrid>, we can do so with its rowClasses attribute, which works the same way that the columnClasses works for columns. Notice the <h:outputStylesheet> tag inside <h:head> near the top of the page, this is a new tag that was introduced in JSF 2.0. One new feature that JSF 2.0 brings to the table is standard resource directories. Resources such as CSS stylesheets, JavaScript files, images, and so on, can be placed under a top level directory named resources, and JSF tags will have access to those resources automatically. In our NetBeans project, we need to place the resources directory under the Web Pages folder. We then need to create a subdirectory to hold our CSS stylesheet (by convention, this directory should be named css), then we place our CSS stylesheet(s) on this subdirectory. The value of the library attribute in <h:outputStylesheet> must match the directory where our CSS file is located, and the value of its name attribute must match the CSS file name. In addition to CSS files, we should place any JavaScript files in a subdirectory called javascript under the resources directory. The file can then be accessed by the <h:outputScript> tag using "javascript" as the value of its library attribute and the file name as the value of its name attribute. Similarly, images should be placed in a directory called images under the resources directory. These images can then be accessed by the JSF <h:graphicImage> tag, where the value of its library attribute would be "images" and the value of its name attribute would be the corresponding file name. Now that we have discussed how to lay out elements on the page and how to access resources, let's focus our attention on the input and output elements on the page. The <h:outputLabel> tag generates a label for an input field in the form, the value of its for attribute must match the value of the id attribute of the corresponding input field. <h:message> generates an error message for an input field, the value of its for field must match the value of the id attribute for the corresponding input field. The first row in our grid contains an <h:selectOneMenu>. This tag generates an HTML <select> tag on the rendered page. Every JSF tag has an id attribute, the value for this attribute must be a string containing a unique identifier for the tag. If we don't specify a value for this attribute, one will be generated automatically. It is a good idea to explicitly state the ID of every component, since this ID is used in runtime error messages. Affected components are a lot easier to identify if we explicitly set their IDs. When using <h:label> tags to generate labels for input fields, or when using <h:message> tags to generate validation errors, we need to explicitly set the value of the id tag, since we need to specify it as the value of the for attribute of the corresponding <h:label> and <h:message> tags. Every JSF input tag has a label attribute. This attribute is used to generate validation error messages on the rendered page. If we don't specify a value for the label attribute, then the field will be identified in the error message by its ID. Each JSF input field has a value attribute, in the case of <h:selectOneMenu>, this attribute indicates which of the options in the rendered <select> tag will be selected. The value of this attribute must match the value of the itemValue attribute of one of the nested <f:selectItem> tags. The value of this attribute is usually a value binding expression, that means that the value is read at runtime from a JSF managed bean. In our example, the value binding expression #{registrationBean.salutation} is used. What will happen is at runtime JSF will look for a managed bean named registrationBean, and look for an attribute named salutation on this bean, the getter method for this attribute will be invoked, and its return value will be used to determine the selected value of the rendered HTML <select> tag. Nested inside the <h:selectOneMenu> there are a number of <f:selectItem> tags. These tags generate HTML <option> tags inside the HTML <select> tag generated by <h:selectOneMenu>. The value of the itemLabel attribute is the value that the user will see while the value of the itemValue attribute will be the value that will be sent to the server when the form is submitted. All other rows in our grid contain <h:inputText> tags, this tag generates an HTML input field of type text, which accept a single line of typed text as input. We explicitly set the id attribute of all of our <h:inputText> fields, this allows us to refer to them from the corresponding <h:outputLabel> and <h:message> fields. We also set the label attribute for all of our <h:inputText> tags, this results in more user-friendly error messages. Some of our <h:inputText> fields require a value, these fields have their required attribute set to true, each JSF input field has a required attribute, if we need to require the user to enter a value for this attribute, then we need to set this attribute to true. This attribute is optional, if we don't explicitly set a value for it, then it defaults to false. In the last row of our grid, we added an empty <h:panelGroup> tag. The purpose of this tag is to allow adding several tags into a single cell of an <h:panelGrid>. Any tags placed inside this tag are placed inside the same cell of the grid where <h:panelGrid> is placed. In this particular case, all we want to do is to have an "empty" cell in the grid so that the next tag, <h:commandButton>, is aligned with the input fields in the rendered page. <h:commandButton> is used to submit a form to the server. The value of its value attribute is used to generate the text of the rendered button. The value of its action attribute is used to determine what page to display after the button is pressed. In our example, we are using static navigation. When using JSF static navigation, the value of the action attribute of a command button is hard-coded in the markup. When using static navigation, the value of the action attribute of <h:commandButton> corresponds to the name of the page we want to navigate to, minus its .xhtml extension. In our example, when the user clicks on the button, we want to navigate to a file named confirmation.xhtml, therefore we used a value of "confirmation" for its action attribute. An alternative to static navigation is dynamic navigation. When using dynamic navigation, the value of the action attribute of the command button is a value binding expression resolving to a method returning a String in a managed bean. The method may then return different values based on certain conditions. Navigation would then proceed to a different page depending on the value of the method. As long as it returns a String, the managed bean method executed when using dynamic navigation can contain any logic inside it, and is frequently used to save data in a managed bean into a database. When using dynamic navigation, the return value of the method executed when clicking the button must match the name of the page we want to navigate to (again, minus the file extension). In earlier versions of JSF, it was necessary to specify navigation rules in facesconfig.xml, with the introduction of the conventions introduced in the previous paragraphs, this is no longer necessary.  
Read more
  • 0
  • 0
  • 7488

article-image-creating-sample-web-scraper
Packt
11 Sep 2013
10 min read
Save for later

Creating a sample web scraper

Packt
11 Sep 2013
10 min read
(For more resources related to this topic, see here.) As web scrapers like to say: "Every website has an API. Sometimes it's just poorly documented and difficult to use." To say that web scraping is a useful skill is an understatement. Whether you're satisfying a curiosity by writing a quick script in an afternoon or building the next Google, the ability to grab any online data, in any amount, in any format, while choosing how you want to store it and retrieve it, is a vital part of any good programmer's toolkit. By virtue of reading this article, I assume that you already have some idea of what web scraping entails, and perhaps have specific motivations for using Java to do it. Regardless, it is important to cover some basic definitions and principles. Very rarely does one hear about web scraping in Java—a language often thought to be solely the domain of enterprise software and interactive web apps of the 90's and early 2000's. However, there are many reasons why Java is an often underrated language for web scraping: Java's excellent exception-handling lets you compile code that elegantly handles the often-messy Web Reusable data structures allow you to write once and use everywhere with ease and safety Java's concurrency libraries allow you to write code that can process other data while waiting for servers to return information (the slowest part of any scraper) The Web is big and slow, but the Java RMI allows you to write code across a distributed network of machines, in order to collect and process data quickly There are a variety of standard libraries for getting data from servers, as well as third-party libraries for parsing this data, and even executing JavaScript (which is needed for scraping some websites) In this article, we will explore these, and other benefits of Java in web scraping, and build several scrapers ourselves. Although it is possible, and recommended, to skip to the sections you already have a good grasp of, keep in mind that some sections build up the code and concepts of other sections. When this happens, it will be noted in the beginning of the section. How is this legal? Web scraping has always had a "gray hat" reputation. While websites are generally meant to be viewed by actual humans sitting at a computer, web scrapers find ways to subvert that. While APIs are designed to accept obviously computer-generated requests, web scrapers must find ways to imitate humans, often by modifying headers, forging POST requests and other methods. Web scraping often requires a great deal of problem solving and ingenuity to figure out how to get the data you want. There are often few roadmaps or tried-and-true procedures to follow, and you must carefully tailor the code to each website—often riding between the lines of what is intended and what is possible. Although this sort of hacking can be fun and challenging, you have to be careful to follow the rules. Like many technology fields, the legal precedent for web scraping is scant. A good rule of thumb to keep yourself out of trouble is to always follow the terms of use and copyright documents on websites that you scrape (if any). There are some cases in which the act of web crawling is itself in murky legal territory, regardless of how the data is used. Crawling is often prohibited in the terms of service of websites where the aggregated data is valuable (for example, a site that contains a directory of personal addresses in the United States), or where a commercial or rate-limited API is available. Twitter, for example, explicitly prohibits web scraping (at least of any actual tweet data) in its terms of service: "crawling the Services is permissible if done in accordance with the provisions of the robots.txt file, however, scraping the Services without the prior consent of Twitter is expressly prohibited" Unless explicitly prohibited by the terms of service, there is no fundamental difference between accessing a website (and its information) via a browser, and accessing it via an automated script. The robots.txt file alone has not been shown to be legally binding, although in many cases the terms of service can be. Writing a simple scraper (Simple) Wikipedia is not just a helpful resource for researching or looking up information but also a very interesting website to scrape. They make no efforts to prevent scrapers from accessing the site, and, with a very well-marked-up HTML, they make it very easy to find the information you're looking for. In this project, we will scrape an article from Wikipedia and retrieve the first few lines of text from the body of the article. Getting ready It is recommended that you have some working knowledge of Java, and the ability to create and execute Java programs at this point. As an example, we will use the article from the following Wikipedia link: http://en.wikipedia.org/wiki/Java Note that this article is about the Indonesian island nation Java, not the programming language. Regardless, it seems appropriate to use it as a test subject. We will be using the jsoup library to parse HTML data from websites and convert it into a manipulatable object, with traversable, indexed values (much like an array). In this xercise, we will show you how to download, install, and use Java libraries. In addition, we'll also be covering some of the basics of the jsoup library in particular. How to do it... Now that we're starting to get into writing scrapers, let's create a new project to keep them all bundled together. Carry out the following steps for this task: Open Eclipse and create a new Java project called Scraper. Packages are still considered to be handy for bundling collections of classes together within a single project (projects contain multiple packages, and packages contain multiple classes). You can create a new package by highlighting the Scraper project in Eclipse and going to File | New | Package . By convention, in order to prevent programmers from creating packages with the same name (and causing namespace problems), packages are named starting with the reverse of your domain name (for example, com.mydomain.mypackagename). For the rest of the article, we will begin all our packages with com.packtpub.JavaScraping appended with the package name. Let's create a new package called com.packtpub.JavaScraping.SimpleScraper. Create a new class, WikiScraper, inside the src folder of the package. Download the jsoup core library, the first link, from the following URL: http://jsoup.org/download Place the .jar file you downloaded into the lib folder of the package you just created. In Eclipse, right-click in the Package Explorer window and select Refresh. This will allow Eclipse to update the Package Explorer to the current state of the workspace folder. Eclipse should show your jsoup-1.7.2.jar file (this file may have a different name depending on the version you're using) in the Package Explorer window. Right-click on the jsoup JAR file and select Build Path | Add to Build Path. In your WikiScraper class file, write the following code: package com.packtpub.JavaScraping.SimpleScraper; import org.jsoup.Jsoup; import org.jsoup.nodes.Document; import java.net.*; import java.io.*; public class WikiScraper { public static void main(String[] args) { scrapeTopic("/wiki/Python"); } public static void scrapeTopic(String url){ String html = getUrl("http://www.wikipedia.org/"+url); Document doc = Jsoup.parse(html); String contentText = doc.select("#mw-content-text > p").first().text(); System.out.println(contentText); } public static String getUrl(String url){ URL urlObj = null; try{ urlObj = new URL(url); } catch(MalformedURLException e){ System.out.println("The url was malformed!"); return ""; } URLConnection urlCon = null; BufferedReader in = null; String outputText = ""; try{ urlCon = urlObj.openConnection(); in = new BufferedReader(new InputStreamReader(urlCon.getInputStream())); String line = ""; while((line = in.readLine()) != null){ outputText += line; } in.close(); }catch(IOException e){ System.out.println("There was an error connecting to the URL"); return ""; } return outputText; } } Assuming you're connected to the internet, this should compile and run with no errors, and print the first paragraph of text from the article. How it works... Unlike our HelloWorld example, there are a number of libraries needed to make this code work. We can incorporate all of these using the import statements before the class declaration. There are a number of jsoup objects needed, along with two Java libraries, java.io and java.net , which are needed for creating the connection and retrieving information from the Web. As always, our program starts out in the main method of the class. This method calls the scrapeTopic method, which will eventually print the data that we are looking for (the first paragraph of text in the Wikipedia article) to the screen. scrapeTopic requires another method, getURL, in order to do this. getUrl is a function that we will be using throughout the article. It takes in an arbitrary URL and returns the raw source code as a string. Essentially, it creates a Java URL object from the URL string, and calls the openConnection method on that URL object. The openConnection method returns a URLConnection object, which can be used to create a BufferedReader object. BufferedReader objects are designed to read from, potentially very long, streams of text, stopping at a certain size limit, or, very commonly, reading streams one line at a time. Depending on the potential size of the pages you might be reading (or if you're reading from an unknown source), it might be important to set a buffer size here. To simplify this exercise, however, we will continue to read as long as Java is able to. The while loop here retrieves the text from the BufferedReader object one line at a time and adds it to outputText, which is then returned. After the getURL method has returned the HTML string to scrapeTopic, jsoup is used. jsoup is a Java library that turns HTML strings (such as the string returned by our scraper) into more accessible objects. There are many ways to access and manipulate these objects, but the function you'll likely find yourself using most often is the select function. The jsoup select function returns a jsoup object (or an array of objects, if more than one element matches the argument to the select function), which can be further manipulated, or turned into text, and printed. The crux of our script can be found in this line: String contentText = doc.select("#mw-content-text > p").first().text(); This finds all the elements that match #mw-content-text > p (that is, all p elements that are the children of the elements with the CSS ID mw-content-text), selects the first element of this set, and turns the resulting object into plain text (stripping out all tags, such as <a> tags or other formatting that might be in the text). The program ends by printing this line out to the console. There's more... Jsoup is a powerful library that we will be working with in many applications throughout this article. For uses that are not covered in the article, I encourage you to read the complete documentation at http://jsoup.org/apidocs/. What if you find yourself working on a project where jsoup's capabilities aren't quite meeting your needs? There are literally dozens of Java-based HTML parsers on the market. Summary Thus in this article we took the first step towards web scraping with Java, and also learned how to scrape an article from Wikipedia and retrieve the first few lines of text from the body of the article. Resources for Article : Further resources on this subject: Making a simple cURL request (Simple) [Article] Web Scraping with Python [Article] Generating Content in WordPress Top Plugins—A Sequel [Article]  
Read more
  • 0
  • 0
  • 7485
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at £15.99/month. Cancel anytime
article-image-managing-files-folders-and-registry-items-using-powershell
Packt
23 Apr 2015
27 min read
Save for later

Managing Files, Folders, and Registry Items Using PowerShell

Packt
23 Apr 2015
27 min read
This article is written by Brenton J.W. Blawat, the author of Mastering Windows PowerShell Scripting. When you are automating tasks on servers and workstations, you will frequently run into situations where you need to manage files, folders, and registry items. PowerShell provides a wide variety of cmdlets that enable you to create, view, modify, and delete items on a system. (For more resources related to this topic, see here.) In this article, you will learn many techniques to interact with files, folders, and registry items. These techniques and items include: Registry provider Creating files, folders, registry keys, and registry named values Adding named values to registry keys Verifying the existence of item files, folders, and registry keys Renaming files, folders, registry keys, and named values Copying and moving files and folders Deleting files, folders, registry keys, and named values To properly follow the examples in this article, you will need to sequentially execute the examples. Each example builds on the previous examples, and some of these examples may not function properly if you do not execute the previous steps. Registry provider When you're working with the registry, PowerShell interprets the registry in the same way it does files and folders. In fact, the cmdlets that you use for files and folders are the same that you would use for registry items. The only difference with the registry is the way in which you call the registry path locations. When you want to reference the registry in PowerShell, you use the [RegistryLocation]:Path syntax. This is made available through the PowerShell Windows Registry Provider. While referencing [RegistryLocation]:Path, PowerShell provides you with the ability to use registry abbreviations pertaining to registry path locations. Instead of referencing the full path of HKEY_LOCAL_MACHINE, you can use the abbreviation of HKLM. Some other abbreviations include: HKLM: Abbreviation for HKEY_LOCAL_MACHINE hive HKCU: Abbreviation for HKEY_CURRENT_USER hive HKU: Abbreviation for HKEY_USERS hive HKCR: Abbreviation for HKEY_CLASSES_ROOT hive HKCC: Abbreviation for HKEY_CURRENT_CONFIG hive For example, if you wanted to reference the named values in the Run registry key for programs that start up on boot, the command line syntax would look like this: HKLM:SoftwareMicrosoftWindowsCurrentVersionRun While it is recommended that you don't use cmdlet aliases in your scripts, it is recommended, and a common practice, to use registry abbreviations in your code. This not only reduces the amount of effort to create the scripts but also makes it easier for others to read the registry locations. Creating files, folders, and registry items with PowerShell When you want to create a new file, folder, or registry key, you will need to leverage the new-item cmdlet. The syntax of this command is new-item, calling the –path argument to specify the location, calling the -name argument to provide a name for the item, and the -ItemType argument to designate whether you want a file or a directory (folder). When you are creating a file, it has an additional argument of –value, which allows you to prepopulate data into the file after creation. When you are creating a new registry key in PowerShell, you can omit the –ItemType argument as it is not needed for registry key creation. PowerShell assumes that when you are interacting with the registry using new-item, you are creating registry keys. The new-item command accepts the -force argument in the instance that the file, folder, or key is being created in a space that is restricted by User Account Control (UAC). To create a new folder and registry item, do the following action: New-item –path "c:Program Files" -name MyCustomSoftware –ItemType Directory New-item –path HKCU:SoftwareMyCustomSoftware -force The output of this is shown in the following screenshot: The preceding example shows how you can create folders and registry keys for a custom application. You first create a new folder in c:Program Files named MyCustomSoftware. You then create a new registry key in HKEY_CURRENT_USER:Software named MyCustomSoftware. You start by issuing the new-item cmdlet followed by the –path argument to designate that the new folder should be placed in c:Program Files. You then call the –name argument to specify the name of MyCustomSoftware. Finally, you tell the cmdlet that the -ItemType argument is Directory. After executing this command you will see a new folder in c:Progam Files named MyCustomSoftware. You then create the new registry key by calling the new-item cmdlet and issuing the –path argument and then specifying the HKCU:SoftwareMyCustomSoftware key location, and you complete it with the –force argument to force the creation of the key. After executing this command, you will see a new registry key in HKEY_CURRENT_USER:Software named MyCustomSoftware. One of the main benefits of PowerShell breaking apart the -path, -name, and -values arguments is that you have the ability to customize each of the values before you use them with the new-item cmdlet. For example, if you want to name a log file with the date stamp, add that parameter into a string and set the –name value to a string. To create a log file with a date included in the filename, do the following action: $logpath = "c:Program FilesMyCustomSoftwareLogs" New-item –path $logpath –ItemType Directory | out-null $itemname = (get-date –format "yyyyMMddmmss") + "MyLogFile.txt" $itemvalue = "Starting Logging at: " + " " + (get-date) New-item –path $logpath -name $itemname –ItemType File –value $itemvalue $logfile = $logpath + $itemname $logfile The output of this is shown in the following screenshot: The content of the log file is shown in the following screenshot: The preceding example displays how you can properly create a new log file with a date time path included in the log file name. It also shows how to create a new directory for the logs. It then displays how to include text inside the log file, designating the start of a new log file. Finally, this example displays how you can save the log file name and path in a variable to use later in your scripts. You first start by declaring the path of c:Program FilesMyCustomSoftwareLogs in the $logpath variable. You then use the new-item cmdlet to create a new folder in c:Program FilesMyCustomSoftware named Logs. By piping the command to out-null, the default output of the directory creation is silenced. You then declare the name that you want the file to be by using the get-date cmdlet, with the –format argument set to yyyyMMddmmss, and by adding mylogfile.txt. This will generate a date time stamp in the format of 4 digits including year, month, day, minutes, seconds, and mylogfile.txt. You then set the name of the file to the $itemname variable. Finally, you declare the $itemvalue variable which contains Starting Log at: and the standard PowerShell date time information. After the variables are populated, you issue the new-item command, the –path argument referencing the $logpath variable, the –name argument referencing the $itemname variable, the –ItemType referencing File, and the –value argument referencing the $itemvalue variable. At the end of the script, you will take the $logpath and $itemname variables to create a new variable of $logfile, which contains the location of the log file. As you will see from this example, after you execute the script the log file is populated with the value of Starting Logging at: 03/16/2015 14:38:24. Adding named values to registry keys When you are interacting with the registry, you typically view and edit named values or properties that are contained with in the keys and sub-keys. PowerShell uses several cmdlets to interact with named values. The first is the get-itemproperty cmdlet which allows you to retrieve the properties of a named value. The proper syntax for this cmdlet is to specify get-itemproperty to use the –path argument to specify the location in the registry, and to use the –name argument to specify the named value. The second cmdlet is new-itemproperty, which allows you to create new named values. The proper syntax for this cmdlet is specifying new-itemproperty, followed by the –path argument and the location where you want to create the new named value. You then specify the –name argument and the name you want to call the named value with. Finally, you use the –PropertyType argument which allows you to specify what kind of registry named value you want to create. The PropertyType argument can be set to Binary, DWord, ExpandString, MultiString, String, and Qword, depending on what your need for the registry value is. Finally, you specifythe –value argument which enables you to place a value into that named value. You may also use the –force overload to force the creation of the key in the instance that the key may be restricted by UAC. To create a named value in the registry, do the following action: $regpath = "HKCU:SoftwareMyCustomSoftware" $regname = "BuildTime" $regvalue = "Build Started At: " + " " + (get-date) New-itemproperty –path $regpath –name $regname –PropertyType String –value $regvalue $verifyValue = Get-itemproperty –path $regpath –name $regname Write-Host "The $regName named value is set to: " $verifyValue.$regname The output of this is shown in the following screenshot:   After executing the script, the registry will look like the following screenshot:   This script displays how you can create a registry named value in a specific location. It also displays how you can retrieve a value and display it in the console. You first start by defining several variables. The first variable $regpath defines where you want to create the new named value which is in the HKCU:SoftwareMyCustomSoftware registry key. The second variable $regname defines what you want the new named value to be named, which is BuildTime. The third variable defines what you want the value of the named value to be, which is Build Started At: with the current date and time. The next step in the script is to create the new value. You first call the new-itemproperty cmdlet with the –path argument and specify the $regpath variable. You then use the –name argument and specify $regname. This is followed by specifying the –PropertyType argument and by specifying the string PropertyType. Finally, you specify the –value argument and use the $regvalue variable to fill the named value with data. Proceeding forward in the script, you verify that the named value has proper data by leveraging the get-itemproperty cmdlet. You first define the $verifyvalue variable that captures the data from the cmdlet. You then issue get-itemproperty with the –path argument of $regpath and the –name argument of $regname. You then write to the console that the $regname named value is set to $verifyvalue.$regname. When you are done with script execution, you should have a new registry named value of BuildTime in the HKEY_CURRENT_USER:SoftwareMyCustomSoftware key with a value similar to Build Started At: 03/16/2015 14:49:22. Verifying files, folders, and registry items When you are creating and modifying objects, it's important to make sure that the file, folder, and registry items don't exist prior to creating and modifying them. The test-path cmdlet allows you to test to see if a file, folder, or registry item exists prior to working with it. The proper syntax for this is first calling test-path and then specifying a file, folder, or registry location. The result of the test-path command is True if the object exists or False if the object doesn't exist. To verify if files, folders, and registry entries exist, do the following action: $testfolder = test-path "c:Program FilesMyCustomSoftwareLogs" #Update The Following Line with the Date/Timestamp of your file $testfile = test-path "c:Program FilesMyCustomSoftwareLogs201503163824MyLogFile.txt" $testreg = test-path "HKCU:SoftwareMyCustomSoftware" If ($testfolder) { write-host "Folder Found!" } If ($testfile) { write-host "File Found!" } If ($testreg) { write-host "Registry Key Found!" } The output is shown in the following screenshot: This example displays how to verify if a file, folder, and registry item exists. You first start by declaring a variable to catch the output from the test-path cmdlet. You then specify test-path, followed by a file, folder, or registry item whose existence you want to verify. In this example, you start by using the test-path cmdlet to verify if the Logs folder is located in the c:Program FilesMyCustomSoftware directory. You then store the result in the $testfolder variable. You then use the test-path cmdlet to check if the file located at c:Program FilesMyCustomSoftwareLogs201503163824MyLogFile.txt exists. You then store the result in the $testfile variable. Finally, you use the test-path cmdlet to see if the registry key of HKCU:SoftwareMyCustomSoftware exists. You then store the result in the $testreg variable. To evaluate the variables, you create IF statements to check whether the variables are True and write to the console if the items are found. After executing the script, the console will output the messages Folder Found!, File Found!, and Registry Key Found!. Copying and moving files and folders When you are working in the operating system, there may be instances where you need to copy or move files and folders around on the operating system. PowerShell provides two cmdlets to copy and move files. The copy-item cmdlet allows you to copy a file or a folder from one location to another. The proper syntax of this cmdlet is calling copy-item, followed by –path argument for the source you want to copy and the –destination argument for the destination of the file or folder. The copy-item cmdlet also has the –force argument to write over a read-only or hidden file. There are instances when read-only files cannot be overwritten, such as a lack of user permissions, which will require additional code to change the file attributes before copying over files or folders. The copy-item cmdlet also has a –recurse argument, which allows you to recursively copy the files in a folder and its subdirectories. A common trick to use with the copy-item cmdlet is to rename during the copy operation. To do this, you change the destination to the desired name you want the file or folder to be. After executing the command, the file or folder will have a new name in its destination. This reduces the number of steps required to copy and rename a file or folder. The move-item cmdlet allows you to move files from one location to another. The move-item cmdlet has the same syntax as the copy-item cmdlet. The proper syntax of this cmdlet is calling move-item, followed by the –path argument for the source you want to move and the –destination argument for the destination of the file or folder. The move-item cmdlet also has the –force overload to write over a read-only or hidden file. There are also instances when read-only files cannot be overwritten, such as a lack of user permissions, which will require additional code to change the file attributes before moving files or folders. The move-item cmdlet does not, however, have a -recurse argument. Also, it's important to remember that the move-item cmdlet requires the destination to be created prior to the move. If the destination folder is not available, it will throw an exception. It's recommended to use the test-path cmdlet in conjunction with the move-item cmdlet to verify that the destination exists prior to the move operation. PowerShell has the same file and folder limitations as the core operating system it is being run on. This means that file paths that are longer than 256 characters in length will receive an error message during the copy process. For paths that are over 256 characters in length, you need to leverage robocopy.exe or a similar file copy program to copy or move files. All move-item operations are recursive by default. You do not have to specify the –recurse argument to recursively move files. To copy files recursively, you need to specify the –recurse argument. To copy and move files and folders, do the following action: New-item –path "c:Program FilesMyCustomSoftwareAppTesting" –ItemType Directory | Out-null New-item –path "c:Program FilesMyCustomSoftwareAppTestingHelp" -ItemType Directory | Out-null New-item –path "c:Program FilesMyCustomSoftwareAppTesting" –name AppTest.txt –ItemType File | out-null New-item –path "c:Program FilesMyCustomSoftwareAppTestingHelp" –name HelpInformation.txt –ItemType File | out-null New-item –path "c:Program FilesMyCustomSoftware" -name ConfigFile.txt –ItemType File | out-null move-item –path "c:Program FilesMyCustomSoftwareAppTesting" –destination "c:Program FilesMyCustomSoftwareArchive" –force copy-item –path "c:Program FilesMyCustomSoftwareConfigFile.txt" "c:Program FilesMyCustomSoftwareArchiveArchived_ConfigFile.txt" –force The output of this is shown in the following screenshot: This example displays how to properly use the copy-item and move-item cmdlets. You first start by using the new-item cmdlet with the –path argument set to c:Program FilesMyCustomSoftwareAppTesting and the –ItemType argument set to Directory. You then pipe the command to out-null to suppress the default output. This creates the AppTesting sub directory in the c:Program FilesMyCustomSoftware directory. You then create a second folder using the new-item cmdlet with the –path argument set to c:Program FilesMyCustomSoftwareAppTestingHelp and the –ItemType argument set to Directory. You then pipe the command to out-null. This creates the Help sub directory in the c:Program FilesMyCustomSoftwareAppTesting directory. After creating the directories, you create a new file using the new-item cmdlet with the path of c:Program FilesMyCustomSoftwareAppTesting, the -name argument set to AppTest.txt, the -ItemType argument set to File; you then pipe it to out-null. You create a second file by using the new-item cmdlet with the path of c:Program FilesMyCustomSoftwareAppTestingHelp, the -name argument set to HelpInformation.txt and the -ItemType argument set to File, and then piping it to out-null. Finally, you create a third file using the new-item cmdlet with the path of c:Program FilesMyCustomSoftware, the -name argument set to ConfigFile.txt and the -ItemType argument set to File, and then pipe it to out-null. After creating the files, you are ready to start copying and moving files. You first move the AppTesting directory to the Archive directory by using the move-item cmdlet and then specifying the –path argument with the value of c:Program FilesMyCustomSoftwareAppTesting as the source, the –destination argument with the value of c:Program FilesMyCustomSoftwareArchive as the destination, and the –force argument to force the move if the directory is hidden. You then copy a configuration file by using the copy-item cmdlet, using the –path argument with c:Program FilesMyCustomSoftwareConfigFile.txt as the source, and then specifying the –destination argument with c:Program FilesMyCustomSoftwareArchiveArchived_ConfigFile.txt as the new destination with a new filename;, you then leverage the –force argument to force the copy if the file is hidden. This follow screenshot displays the file folder hierarchy after executing this script: After executing this script, the file folder hierarchy should be as displayed in the preceding screenshot. This also displays that when you move the AppTesting directory to the Archive folder, it automatically performs the move recursively, keeping the file and folder structure intact. Renaming files, folders, registry keys, and named values When you are working with PowerShell scripts, you may have instances where you need to rename files, folders, and registry keys. The rename-item cmdlet can be used to perform renaming operations on a system. The syntax for this cmdlet is rename-item and specifying the –path argument with path to the original object, and then you call the –newname argument with a full path to what you want the item to be renamed to. The rename-item has a –force argument to force the rename in instances where the file or folder is hidden or restricted by UAC or to avoid prompting for the rename action. To copy and rename files and folders, do the following action: New-item –path "c:Program FilesMyCustomSoftwareOldConfigFiles" –ItemType Directory | out-null Rename-item –path "c:Program FilesMyCustomSoftwareOldConfigFiles" –newname "c:Program FilesMyCustomSoftwareConfigArchive" -force copy-item –path "c:Program FilesMyCustomSoftwareConfigFile.txt" "c:Program FilesMyCustomSoftwareConfigArchiveConfigFile.txt" –force Rename-item –path "c:Program FilesMyCustomSoftwareConfigArchiveConfigFile.txt" –newname "c:Program FilesMyCustomSoftwareConfigArchiveOld_ConfigFile.txt" –force The output of this is shown in the following screenshot: In this example, you create a script that creates a new folder and a new file, and then renames the file and the folder. To start, you leverage the new-item cmdlet which creates a new folder in c:Program FilesMyCustomSoftware named OldConfigFiles. You then pipe that command to Out-Null, which silences the standard console output of the folder creation. You proceed to rename the folder c:Program FilesMyCustomSoftwareOldConfigFiles with the rename-item cmdlet using the –newname argument to c:Program FilesMyCustomSoftwareConfigArchive. You follow the command with the –force argument to force the renaming of the folder. You leverage the copy-item cmdlet to copy the ConfigFile.txt into the ConfigArchive directory. You first start by specifying the copy-item cmdlet with the –path argument set to c:Program FilesMyCustomSoftwareConfigFile.txt and the destination set to c:Program FilesMyCustomSoftwareConfigArchiveConfigFile.txt. You include the –Force argument to force the copy. After moving the file, leverage the rename-item cmdlet with the -path argument to rename c:Program FilesMyCustomSoftwareConfigArchiveConfigFile.txt using the –newname argument to c:Program FilesMyCustomSoftwareConfigArchiveOld_ConfigFile.txt. You follow this command with the –force argument to force the renaming of the file. At the end of this script, you will have successfully renamed a folder, moved a file into that renamed folder, and renamed a file in the newly created folder. In the instance that you want to rename a registry key, do the following action: New-item –path "HKCU:SoftwareMyCustomSoftware" –name CInfo –force | out-null Rename-item –path "HKCU:SoftwareMyCustomSoftwareCInfo" –newname ConnectionInformation –force The output of this is shown in the following screenshot: After renaming the subkey, the registry will look like the following screenshot: This example displays how to create a new subkey and rename it. You first start by using the new-item cmdlet to create a new sub key with the –path argument of the HKCU:SoftwareMyCustomSoftware key and the -name argument set to CInfo. You then pipe that line to out-null in order to suppress the standard output from the script. You proceed to execute the rename-item cmdlet with the –path argument set to HKCU:SoftwareMyCustomSoftwareCInfo and the –newname argument set to ConnectionInformation. You then use the –force argument to force the renaming in instances when the subkey is restricted by UAC. After executing this command, you will see that the CInfo subkey located in HKCU:SoftwareMyCustomSoftware is now renamed to ConnectionInformation. When you want to update named values in the registry, you will not be able to use the rename-item cmdlet. This is due to the named values being properties of the keys themselves. Instead, PowerShell provides the rename-itemproperty cmdlet to rename the named values in the key. The proper syntax for this cmdlet is calling rename-itemproperty by using the –path argument, followed by the path to the key that contains the named value. You then issue the –name argument to specify the named value you want to rename. Finally, you specify the –newname argument and the name you want the named value to be renamed to. To rename the registry named value, do the following action: $regpath = "HKCU:SoftwareMyCustomSoftwareConnectionInformation" $regname = "DBServer" $regvalue = "mySQLserver.mydomain.local" New-itemproperty –path $regpath –name $regname –PropertyType String –value $regvalue | Out-null Rename-itemproperty –path $regpath –name DBServer –newname DatabaseServer The output of this is shown in the following screenshot: After updating the named value, the registry will reflect this change, and so should look like the following screenshot: The preceding script displays how to create a new named value and rename it to a different named value. You first start by defining the variables to be used with the new-itemproperty cmdlet. You define the location of the registry subkey in the $regpath variable and set it to HKCU:SoftwareMyCustomSoftwareConnectionInformation. You then specify the named value name of DBServer and store it in the $regname variable. Finally, you define the $regvalue variable and store the value of mySQLserver.mydomain.local. To create the new named value, leverage new-itemproperty, specify the –path argument with the $regpath variable, use the –name argument with the $regname variable, and use the –value argument with the $regvalue variable. You then pipe this command to out-null in order to suppress the default output of the command. This command will create the new named value of DBServer with the value of mySQLserver.mydomain.local in the HKCU:SoftwareMyCustomSoftwareConnectionInformation subkey. The last step in the script is renaming the DBServer named value to DatabaseServer. You first start by calling the rename-itemproperty cmdlet and then using the –path argument and specifying the $regpath variable which contains the HKCU:SoftwareMyCustomSoftwareConnectionInformation subkey; you then proceed by calling the –name argument and specifying DBServer and finally calling the –newname argument with the new value of DatabaseServer. After executing this command, you will see that the HKCU:SoftwareMyCustomSoftwareConnectionInformation key has a new named value of DatabaseServer containing the same value of mySQLserver.mydomain.local. Deleting files, folders, registry keys, and named values When you are creating scripts, there are instances when you need to delete items from a computer. PowerShell has the remove-item cmdlet that enables the removal of objects from a computer. The syntax of this cmdlet starts by calling the remove-item cmdlet and proceeds with specifying the –path argument with a file, folder, or registry key to delete. The remove-item cmdlet has several useful arguments that can be leveraged. The –force argument is available to delete files, folders, and registry keys that are read-only, hidden, or restricted by UAC. The –recurse argument is available to enable recursive deletion of files, folders, and registry keys on a system. The –include argument enables you to delete specific files, folders, and registry keys. The –include argument allows you to use the wildcard character of an asterisk (*) to search for specific values in an object name or a specific object type. The –exclude argument will exclude specific files, folders, and registry keys on a system. It also accepts the wildcard character of an asterisk (*) to search for specific values in an object name or a specific object type. The named values in the registry are properties of the key that they are contained in. As a result, you cannot use the remove-item cmdlet to remove them. Instead, PowerShell offers the remove-itemproperty cmdlet to enable the removal of the named values. The remove-itemproperty cmdlet has arguments similar to those of the remove-item cmdlet. It is important to note, however, that the –filter, -include, and –exclude arguments will not work with named values in the registry. They only work with item paths such as registry keys. To set up the system for the deletion example, you need to process the following script: # Create New Directory new-item –path "c:program filesMyCustomSoftwareGraphics" –ItemType Directory | Out-null # Create Files for This Example new-item –path "c:program filesMyCustomSoftwareGraphics" –name FirstGraphic.bmp –ItemType File | Out-Null new-item –path "c:program filesMyCustomSoftwareGraphics" –name FirstGraphic.png –ItemType File | Out-Null new-item –path "c:program filesMyCustomSoftwareGraphics" –name SecondGraphic.bmp –ItemType File | Out-Null new-item –path "c:program filesMyCustomSoftwareGraphics" –name SecondGraphic.png –ItemType File | Out-Null new-item –path "c:program filesMyCustomSoftwareLogs" –name 201301010101LogFile.txt –ItemType File | Out-Null new-item –path "c:program filesMyCustomSoftwareLogs" –name 201302010101LogFile.txt –ItemType File | Out-Null new-item –path "c:program filesMyCustomSoftwareLogs" –name 201303010101LogFile.txt –ItemType File | Out-Null # Create New Registry Keys and Named Values New-item –path "HKCU:SoftwareMyCustomSoftwareAppSettings" | Out-null New-item –path "HKCU:SoftwareMyCustomSoftwareApplicationSettings" | Out-null New-itemproperty –path "HKCU:SoftwareMyCustomSoftwareApplicationSettings" –name AlwaysOn –PropertyType String –value True | Out-null New-itemproperty –path "HKCU:SoftwareMyCustomSoftwareApplicationSettings" –name AutoDeleteLogs –PropertyType String –value True | Out-null The output of this is shown in the following screenshot: The preceding example is designed to set up the file structure for the following example. You first use the new-item cmdlet to create a new directory called Graphics in c:program filesMyCustomSoftware. You then use the new-item cmdlet to create new files named FirstGraphic.bmp, FirstGraphic.png, SecondGraphic.bmp, and SecondGraphic.png in the c:Program FilesMyCustomSoftwareGraphics directory. You then use the new-item cmdlet to create new log files in c:Program FilesMyCustomSoftwareLogs named 201301010101LogFile.txt, 201302010101LogFile.txt, and 201303010101LogFile.txt. After creating the files, you create two new registry keys located at HKCU:SoftwareMyCustomSoftwareAppSettings and HKCU:SoftwareMyCustomSoftwareApplicationSettings. You then populate the HKCU:SoftwareMyCustomSoftwareApplicationSettings key with a named value of AlwaysOn set to True and a named value of AutoDeleteLogs set to True. To remove files, folders, and registry items from a system, do the following action: # Get Current year $currentyear = get-date –f yyyy # Build the Exclude String $exclude = "*" + $currentyear + "*" # Remove Items from System Remove-item –path "c:Program FilesMyCustomSoftwareGraphics" –include *.bmp –force -recurse Remove-item –path "c:Program FilesMyCustomSoftwareLogs" –exclude $exclude -force –recurse Remove-itemproperty –path "HKCU:SoftwareMyCustomSoftwareApplicationSettings" –Name AutoDeleteLogs Remove-item –path "HKCU:SoftwareMyCustomSoftwareApplicationSettings" The output of this is shown in the following screenshot: This script displays how you can leverage PowerShell to clean up files and folders with the remove-item cmdlet and the –exclude and –include arguments. You first start by building the exclusion string for the remove-item cmdlet. You retrieve the current year by using the get-date cmdlet with the –f parameter set to yyyy. You save the output into the $currentyear variable. You then create a $exclude variable that appends asterisks on each end of the $currentyear variable, which contains the current date. This will allow the exclusion filter to find the year anywhere in the file or folder names. The first command is that you use the remove-item cmdlet and call the –path argument with the path of c:Program FilesMyCustomSoftwareGraphics. You then specify the –include argument with the value of *.bmp. This tells the remove-item cmdlet to delete all files that end in .bmp. You then specify –force to force the deletion of the files and –recurse to search the entire Graphics directory to delete the files that meet the *.bmp inclusion criteria but leaves the other files you created with the *.png extension. The second command leverages the remove-item cmdlet with the –path argument set to c:Program FilesMyCustomSoftwareLogs. You use the –exclude argument with the value of $exclude to exclude files that contain the current year. You then specify –force to force the deletion of the files and –recurse to search the entire logs directory to delete the files and folders that do not meet the exclusion criteria. The third command leverages the remove-itemproperty cmdlet with the –path argument set to HKCU:SoftwareMyCustomSoftwareApplicationSettings and the –name argument set to AutoDeleteLogs. After execution, the AutoDeleteLogs named path is deleted from the registry. The last command leverages the remote-item cmdlet with the –path argument set to HKCU:SoftwareMyCustomSoftwareApplicationSettings. After running this last command, the entire subkey of ApplicationSettings is removed from HKCU:SoftwareMyCustomSoftware. After executing this script, you will see that the script deletes the .BMP files in the c:Program FilesMyCustomSoftwareGraphics directory, but it leaves the .PNG files. You will also see that the script deletes all of the log files except the ones that had the current year contained in them. Last, you will see that the ApplicationSettings sub key that was created in the previous step is successfully deleted from HKCU:SoftwareMyCustomSoftware. When you use the remove-item and -recurse parameters together, it is important to note that if remote-item cmdlet deletes all the files and folders in a directory, the –recurse parameter will also delete the empty folder and subfolders that contained those files. This is only true when there are no remaining files in the folders in a particular directory. This may create undesirable results on your system, and so you should use caution while performing this combination. Summary This article thoroughly explained the interaction of PowerShell with the files, folders, and registry objects. It began by displaying how to create a folder and a registry key by leveraging the new-item cmdlet. It also displayed the additional arguments that can be used with the new-item cmdlet to create a log file with the date time integrated in the filename. The article proceeded to display how to create and view a registry key property using the get-itemproperty and new-itemproperty cmdlets This article then moved to verification of files, folder, and registry items through the test-path cmdlet. By using this cmdlet, you can test to see if the object exists prior to interacting with it. You also learned how to interact with copying and moving files and folders by leveraging the copy-item and move-item cmdlets. You also learned how to rename files, folders, registry keys and registry properties with the use of the rename-item and rename-itemproperty cmdlets. This article ends with learning how to delete files, folders, and registry items by leveraging the remove-item and remove-itemproperty cmdlets. Resources for Article: Further resources on this subject: Azure Storage [article] Hyper-V Basics [article] Unleashing Your Development Skills with PowerShell [article]
Read more
  • 0
  • 0
  • 7482

article-image-recording-interview-skype-using-audacity-13
Packt
23 Mar 2010
4 min read
Save for later

Recording an Interview with Skype using Audacity 1.3

Packt
23 Mar 2010
4 min read
In a previous article we described everything that you need to know about recording voice tracks. In this article by Bethany Hiitola, author of Getting started with Audacity 1.3, we will learn all the details of using third-party internet telephony software such as Skype to record telephone interviews. We will also cover how to set up a timed recording. Recording an interview with Skype If you are interested in doing more than solo podcasts with Audacity, you can always try creating interview podcasts. You can record these live in your office with your computer's internal microphone, or with additional microphones. However, you can't always perform an interview from the comfort of your office due to conflicting schedules and the location of your interviewee. Hence, let's learn how to record an interview using your phone and your computer. First you'll need to install another software that allows you to make phone calls using your computer. The program we are going to use for this example is Skype. However, you could use other software that does the same thing for your Internet telephony set-up. Download and install Skype Skype is software that allows us to make voice calls over the Internet, particularly to other users of Skype. Some numbers (such as toll-free numbers) are free of charge, while calls to landlines and mobile phones may require a small fee. For details on pricing for Skype credits for landline and cell calls go to: http://www.skype.com/. Let's briefy discuss how to download and install Skype. First, go to http://www.skype.com/ and download the appropriate version of the software for your computer. Once the installation package has been downloaded to your computer, double-click on it to begin the installation. For Mac computers, a .DMG file is downloaded. All you need to do is uncompress that file and drag-and-drop the Skype package to the Application folder. For any Windows device, an .exe file is downloaded. Double-click on that file to begin the installation. For Linux, there are multiple distributions available. If you aren't already prompted to do so, start the Skype application and follow the on-screen instructions to sign up for a new Skype account. Once you have registered and signed in, the main Skype screen is displayed, which should look similar to the next screenshot: Set up Skype for your telephone interview For our project, we've been using the computer's internal microphone, so there shouldn't be any additional set up in either Skype or Audacity. However, to be sure you may want to check the recording input devices in Audacity to make sure that you can record both sides of the interview. To do this, use the following steps: In the Audacity window, go to the main menu, and then select Audacity and then Preferences. When using a computer running the Microsoft Windows or Linux operating systems, you can find these preferences from the main menu. Select File and then Preferences. In the Audacity Preferences window, select Devices. Check the Device settings under Recording. Particularly if you are using multiple inputs, it may be best to select Stereo Mixer or similar input. If there are many devices listed with the Recording | Device drop-down menu, perform a few interview tests with a friend on Skype prior to the recording session, to determine which of the connected devices will actually be doing the recording. You might also want to turn off all notifcations in Skype. These are all of the alert sounds for events such as contacts logging in and out, incoming call alerts, and so on. To do this, follow the steps shown below: Open Skype and log in. From the main menu, select Skype and then Preferences. Select the Notifcations tab. Be sure to uncheck the Play Sound checkbox. This will make sure that all sounds are suspended and won't interrupt your recording session.
Read more
  • 0
  • 0
  • 7477

article-image-preparing-and-automating-a-task-in-python-tutorial
Bhagyashree R
10 Jan 2019
15 min read
Save for later

Preparing and automating a task in Python [Tutorial]

Bhagyashree R
10 Jan 2019
15 min read
To properly automate tasks, we need a platform so that they run automatically at the proper times. A task that needs to be run manually is not really fully automated. But, in order to be able to leave them running in the background while worrying about more pressing issues, the task will need to be adequate to run in fire-and-forget mode. We should be able to monitor that it runs correctly, be sure that we are capturing future actions (such as receiving notifications if something interesting arises), and know whether there have been any errors while running it. Ensuring that a piece of software runs consistently with high reliability is actually a very big deal and is one area that, to be done properly, requires specialized knowledge and staff, which typically go by the names of sysadmin, operations, or SRE (Site Reliability Engineering). In this article, we will learn how to prepare and automatically run tasks. It covers how to program tasks to be executed when they should, instead of running them manually, and how to be notified if there has been an error in an automated process. This article is an excerpt from a book written by Jaime Buelta titled Python Automation Cookbook.  The Python Automation Cookbook helps you develop a clear understanding of how to automate your business processes using Python, including detecting opportunities by scraping the web, analyzing information to generate automatic spreadsheets reports with graphs, and communicating with automatically generated emails. To follow along with the examples implemented in the article, you can find the code on the book's GitHub repository. Preparing a task It all starts with defining exactly what task needs to be run and designing it in a way that doesn't require human intervention to run. Some ideal characteristic points are as follows: Single, clear entry point: No confusion on what the task to run is. Clear parameters: If there are any parameters, they should be very explicit. No interactivity: Stopping the execution to request information from the user is not possible. The result should be stored: To be able to be checked at a different time than when it runs. Clear result: If we are working interactively in a result, we accept more verbose results or progress reports. But, for an automated task, the final result should be as concise and to the point as possible. Errors should be logged: To analyze what went wrong. A command-line program has a lot of those characteristics already. It has a clear way of running, with defined parameters, and the result can be stored, even if just in text format. But, it can be improved with a config file to clarify the parameters and an output file. Getting ready We'll start by following a structure in which the main function will serve as the entry point, and all parameters are supplied to it. The definition of the main function with all the explicit arguments covers points 1 and 2. Point 3 is not difficult to achieve. To improve point 2 and 5, we'll look at retrieving the configuration from a file and storing the result in another. How to do it... Prepare the following task and save it as prepare_task_step1.py: import argparse def main(number, other_number): result = number * other_number print(f'The result is {result}') if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument('-n1', type=int, help='A number', default=1) parser.add_argument('-n2', type=int, help='Another number', default=1) args = parser.parse_args() main(args.n1, args.n2) Update the file to define a config file that contains both arguments, and save it as prepare_task_step2.py: import argparse import configparser def main(number, other_number): result = number * other_number print(f'The result is {result}') if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument('-n1', type=int, help='A number', default=1) parser.add_argument('-n2', type=int, help='Another number', default=1) parser.add_argument('--config', '-c', type=argparse.FileType('r'), help='config file') args = parser.parse_args() if args.config: config = configparser.ConfigParser() config.read_file(args.config) # Transforming values into integers args.n1 = int(config['DEFAULT']['n1']) args.n2 = int(config['DEFAULT']['n2']) main(args.n1, args.n2) Create the config file config.ini: [ARGUMENTS] n1=5 n2=7 Run the command with the config file: $ python3 prepare_task_step2.py -c config.ini The result is 35 $ python3 prepare_task_step2.py -c config.ini -n1 2 -n2 3 The result is 35 Add a parameter to store the result in a file, and save it as prepare_task_step5.py: import argparse import sys import configparser def main(number, other_number, output): result = number * other_number print(f'The result is {result}', file=output) if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument('-n1', type=int, help='A number', default=1) parser.add_argument('-n2', type=int, help='Another number', default=1) parser.add_argument('--config', '-c', type=argparse.FileType('r'), help='config file') parser.add_argument('-o', dest='output', type=argparse.FileType('w'), help='output file', default=sys.stdout) args = parser.parse_args() if args.config: config = configparser.ConfigParser() config.read_file(args.config) # Transforming values into integers args.n1 = int(config['DEFAULT']['n1']) args.n2 = int(config['DEFAULT']['n2']) main(args.n1, args.n2, args.output) Run the result to check that it's sending the output to the defined file: $ python3 prepare_task_step5.py -n1 3 -n2 5 -o result.txt $ cat result.txt The result is 15 $ python3 prepare_task_step5.py -c config.ini -o result2.txt $ cat result2.txt The result is 35 How it works... Note that the argparse module allows us to define files as parameters, with the argparse.FileType type, and opens them automatically. This is very handy and will raise an error if the file is not valid. The configparser module allows us to use config files with ease. As demonstrated in Step 2, the parsing of the file is as simple as follows: config = configparser.ConfigParser() config.read_file(file) The config will then be accessible as a dictionary divided by sections, and then values. Note that the values are always stored in string format, requiring to be transformed into other types, such as integers. Python 3 allows us to pass a file parameter to the print function, which will write to that file. Step 5 shows the usage to redirect all the printed information to a file. Note that the default parameter is sys.stdout, which will print the value to the Terminal (standard output). This makes it so that calling the script without an -o parameter will display the information on the screen, which is helpful in debugging: $ python3 prepare_task_step5.py -c config.ini The result is 35 $ python3 prepare_task_step5.py -c config.ini -o result.txt $ cat result.txt The result is 35 Setting up a cron job Cron is an old-fashioned but reliable way of executing commands. It has been around since the 70s in Unix, and it's an old favorite in system administration to perform maintenance, such as freeing space, rotating logs, making backups, and other common operations. Getting ready We will produce a script, called  cron.py: import argparse import sys from datetime import datetime import configparser def main(number, other_number, output): result = number * other_number print(f'[{datetime.utcnow().isoformat()}] The result is {result}', file=output) if __name__ == '__main__': parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter) parser.add_argument('--config', '-c', type=argparse.FileType('r'), help='config file', default='/etc/automate.ini') parser.add_argument('-o', dest='output', type=argparse.FileType('a'), help='output file', default=sys.stdout) args = parser.parse_args() if args.config: config = configparser.ConfigParser() config.read_file(args.config) # Transforming values into integers args.n1 = int(config['DEFAULT']['n1']) args.n2 = int(config['DEFAULT']['n2']) main(args.n1, args.n2, args.output) Note the following details: The config file is by default, /etc/automate.ini. Reuse config.ini from the previous recipe. A timestamp has been added to the output. This will make it explicit when the task is run. The result is being added to the file, as shown with the 'a' mode where the file is open. The ArgumentDefaultsHelpFormatter parameter automatically adds information about default values when printing the help using the -h argument. Check that the task is producing the expected result and that you can log to a known file: $ python3 cron.py [2018-05-15 22:22:31.436912] The result is 35 $ python3 cron.py -o /path/automate.log $ cat /path/automate.log [2018-05-15 22:28:08.833272] The result is 35 How to do it... Obtain the full path of the Python interpreter. This is the interpreter that's on your virtual environment: $ which python /your/path/.venv/bin/python Prepare the cron to be executed. Get the full path and check that it can be executed with no problem. Execute it a couple of times: $ /your/path/.venv/bin/python /your/path/cron.py -o /path/automate.log $ /your/path/.venv/bin/python /your/path/cron.py -o /path/automate.log Check that the result is being added correctly to the result file: $ cat /path/automate.log [2018-05-15 22:28:08.833272] The result is 35 [2018-05-15 22:28:10.510743] The result is 35 Edit the crontab file to run the task once every five minutes: $ crontab -e */5 * * * * /your/path/.venv/bin/python /your/path/cron.py -o /path/automate.log Note that this opens an editing Terminal with your default command-line editor. Check the crontab contents. Note that this displays the crontab contents, but doesn't set it to edit: $ contab -l */5 * * * * /your/path/.venv/bin/python /your/path/cron.py -o /path/automate.log Wait and check the result file to see how the task is being executed: $ tail -F /path/automate.log [2018-05-17 21:20:00.611540] The result is 35 [2018-05-17 21:25:01.174835] The result is 35 [2018-05-17 21:30:00.886452] The result is 35 How it works... The crontab line consists of a line describing how often to run the task (first six elements), plus the task. Each of the initial six elements mean a different unit of time to execute. Most of them are stars, meaning any: * * * * * * | | | | | | | | | | | +-- Year (range: 1900-3000) | | | | +---- Day of the Week (range: 1-7, 1 standing for Monday) | | | +------ Month of the Year (range: 1-12) | | +-------- Day of the Month (range: 1-31) | +---------- Hour (range: 0-23) +------------ Minute (range: 0-59) Therefore, our line, */5 * * * * *, means every time the minute is divisible by 5, in all hours, all days... all years. Here are some examples: 30 15 * * * * means "every day at 15:30" 30 * * * * * means "every hour, at 30 minutes" 0,30 * * * * * means "every hour, at 0 minutes and 30 minutes" */30 * * * * * means "every half hour" 0 0 * * 1 * means "every Monday at 00:00" Do not try to guess too much. Use a cheat sheet like crontab guru for examples and tweaks. Most of the common usages will be described there directly. You can also edit a formula and get a descriptive text on how it's going to run. After the description of how to run the cron job, including the line to execute the task, as prepared in Step 2 in the How to do it… section. Capturing errors and problems An automated task's main characteristic is its fire-and-forget quality. We are not actively looking at the result, but making it run in the background. This recipe will present an automated task that will safely store unexpected behaviors in a log file that can be checked afterward. Getting ready As a starting point, we'll use a task that will divide two numbers, as described in the command line. How to do it... Create the task_with_error_handling_step1.py file, as follows: import argparse import sys def main(number, other_number, output): result = number / other_number print(f'The result is {result}', file=output) if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument('-n1', type=int, help='A number', default=1) parser.add_argument('-n2', type=int, help='Another number', default=1) parser.add_argument('-o', dest='output', type=argparse.FileType('w'), help='output file', default=sys.stdout) args = parser.parse_args() main(args.n1, args.n2, args.output) Execute it a couple of times to see that it divides two numbers: $ python3 task_with_error_handling_step1.py -n1 3 -n2 2 The result is 1.5 $ python3 task_with_error_handling_step1.py -n1 25 -n2 5 The result is 5.0 Check that dividing by 0 produces an error and that the error is not logged on the result file: $ python task_with_error_handling_step1.py -n1 5 -n2 1 -o result.txt $ cat result.txt The result is 5.0 $ python task_with_error_handling_step1.py -n1 5 -n2 0 -o result.txt Traceback (most recent call last): File "task_with_error_handling_step1.py", line 20, in <module> main(args.n1, args.n2, args.output) File "task_with_error_handling_step1.py", line 6, in main result = number / other_number ZeroDivisionError: division by zero $ cat result.txt Create the task_with_error_handling_step4.py file: import logging import sys import logging LOG_FORMAT = '%(asctime)s %(name)s %(levelname)s %(message)s' LOG_LEVEL = logging.DEBUG def main(number, other_number, output): logging.info(f'Dividing {number} between {other_number}') result = number / other_number print(f'The result is {result}', file=output) if __name__ == '__main__': parser = argparse.ArgumentParser() parser.add_argument('-n1', type=int, help='A number', default=1) parser.add_argument('-n2', type=int, help='Another number', default=1) parser.add_argument('-o', dest='output', type=argparse.FileType('w'), help='output file', default=sys.stdout) parser.add_argument('-l', dest='log', type=str, help='log file', default=None) args = parser.parse_args() if args.log: logging.basicConfig(format=LOG_FORMAT, filename=args.log, level=LOG_LEVEL) else: logging.basicConfig(format=LOG_FORMAT, level=LOG_LEVEL) try: main(args.n1, args.n2, args.output) except Exception as exc: logging.exception("Error running task") exit(1) Run it to check that it displays the proper INFO and ERROR log and that it stores it on the log file: $ python3 task_with_error_handling_step4.py -n1 5 -n2 0 2018-05-19 14:25:28,849 root INFO Dividing 5 between 0 2018-05-19 14:25:28,849 root ERROR division by zero Traceback (most recent call last): File "task_with_error_handling_step4.py", line 31, in <module> main(args.n1, args.n2, args.output) File "task_with_error_handling_step4.py", line 10, in main result = number / other_number ZeroDivisionError: division by zero $ python3 task_with_error_handling_step4.py -n1 5 -n2 0 -l error.log $ python3 task_with_error_handling_step4.py -n1 5 -n2 0 -l error.log $ cat error.log 2018-05-19 14:26:15,376 root INFO Dividing 5 between 0 2018-05-19 14:26:15,376 root ERROR division by zero Traceback (most recent call last): File "task_with_error_handling_step4.py", line 33, in <module> main(args.n1, args.n2, args.output) File "task_with_error_handling_step4.py", line 11, in main result = number / other_number ZeroDivisionError: division by zero 2018-05-19 14:26:19,960 root INFO Dividing 5 between 0 2018-05-19 14:26:19,961 root ERROR division by zero Traceback (most recent call last): File "task_with_error_handling_step4.py", line 33, in <module> main(args.n1, args.n2, args.output) File "task_with_error_handling_step4.py", line 11, in main result = number / other_number ZeroDivisionError: division by zero How it works... To properly capture any unexpected exceptions, the main function should be wrapped into a try-except block, as done in Step 4 in the How to do it… section. Compare this to how Step 1 is not wrapping the code: try: main(...) except Exception as exc: # Something went wrong logging.exception("Error running task") exit(1) The extra step to exit with status 1 with the exit(1) call informs the operating system that something went wrong with our script. The logging module allows us to log. Note the basic configuration, which includes an optional file to store the logs, the format, and the level of the logs to display. Creating logs is easy. You can do this by making a call to the method logging.<logging level>, (where logging level is debug, info, and so on). logging.exception() is a special case that will create an ERROR log, but it will also include information about the exception, such as the stack trace. Remember to check logs to discover errors. A useful reminder is to add a note on the results file, like this: try: main(args.n1, args.n2, args.output) except Exception as exc: logging.exception(exc) print('There has been an error. Check the logs', file=args.output) In this article, we saw how to define and design a task so that no human intervention is needed to run it. We learned how to use cron for automating a task. We further presented an automated task that will safely store unexpected behaviors in a log file that can be checked afterward. If you found this post useful, do check out the book, Python Automation Cookbook to develop a clear understanding of how to automate your business processes using Python. This includes detecting opportunities by scraping the web, analyzing information to generate automatic spreadsheets reports with graphs, and communicating with automatically generated emails. Write your first Gradle build script to start automating your project [Tutorial] Ansible 2 for automating networking tasks on Google Cloud Platform [Tutorial] Automating OpenStack Networking and Security with Ansible 2 [Tutorial]
Read more
  • 0
  • 0
  • 7475

article-image-working-json-php-jquery
Packt
20 Dec 2010
5 min read
Save for later

Working with JSON in PHP jQuery

Packt
20 Dec 2010
5 min read
  PHP jQuery Cookbook Over 60 simple but highly effective recipes to create interactive web applications using PHP with jQuery Create rich and interactive web applications with PHP and jQuery Debug and execute jQuery code on a live site Design interactive forms and menus Another title in the Packt Cookbook range, which will help you get to grips with PHP as well as jQuery         Read more about this book       In this article, by Vijay Joshi, author of PHP jQuery Cookbook, we will cover: Creating JSON in PHP Reading JSON in PHP Catching JSON parsing errors Accessing data from a JSON in jQuery (For more resources on this subject, see here.) Introduction Recently, JSON (JavaScript Object Notation) has become a very popular data interchange format with more and more developers opting for it over XML. Even many web services nowadays provide JSON as the default output format. JSON is a text format that is programming-language independent and is a native data form of JavaScript. It is lighter and faster than XML because it needs less markup compared to XML. Because JSON is the native data form of JavaScript, it can be used on the client side in an AJAX application more easily than XML. A JSON object starts with { and ends with }. According to the JSON specification, the following types are allowed in JSON: Object: An object is a collection of key-value pairs enclosed between { and } and separated by a comma. The key and the value themselves are separated using a colon (:). Think of objects as associative arrays or hash tables. Keys are simple strings and values can be an array, string, number, boolean, or null. Array: Like other languages, an array is an ordered pair of data. For representing an array, values are comma separated and enclosed between [ and ]. String: A string must be enclosed in double quotes The last type is a number A JSON can be as simple as: { "name":"Superman", "address": "anywhere"} An example using an array is as follows: { "name": "Superman", "phoneNumbers": ["8010367150", "9898989898", "1234567890" ]} A more complex example that demonstrates the use of objects, arrays, and values is as follows:   { "people": [ { "name": "Vijay Joshi", "age": 28, "isAdult": true }, { "name": "Charles Simms", "age": 13, "isAdult": false } ]} An important point to note: { 'name': 'Superman', 'address': 'anywhere'} Above is a valid JavaScript object but not a valid JSON. JSON requires that the name and value must be enclosed in double quotes; single quotes are not allowed. Another important thing is to remember the proper charset of data. Remember that JSON expects the data to be UTF-8 whereas PHP adheres to ISO-8859-1 encoding by default. Also note that JSON is not a JavaScript; it is basically a specification or a subset derived from JavaScript. Now that we are familiar with JSON, let us proceed towards the recipes where we will learn how we can use JSON along with PHP and jQuery. Create a new folder and name it as Chapter 4. We will put all the recipes of this article together in this folder. Also put the jquery.js file inside this folder. To be able to use PHP's built-in JSON functions, you should have PHP version 5.2 or higher installed. Creating JSON in PHP This recipe will explain how JSON can be created from PHP arrays and objects Getting ready Create a new folder inside the Chapter4 directory and name it as Recipe1. How to do it... Create a file and save it by the name index.php in the Recipe1 folder. Write the PHP code that creates a JSON string from an array. <?php $travelDetails = array( 'origin' => 'Delhi', 'destination' => 'London', 'passengers' => array ( array('name' => 'Mr. Perry Mason', 'type' => 'Adult', 'age'=> 28), array('name' => 'Miss Irene Adler', 'type' => 'Adult', 'age'=> 28) ), 'travelDate' => '17-Dec-2010' ); echo json_encode($travelDetails);?> Run the file in your browser. It will show a JSON string as output on screen. After indenting the result will look like the following: { "origin":"Delhi","destination":"London","passengers":[ { "name":"Mr. Perry Mason", "type":"Adult", "age":28 }, { "name":"Miss Irene Adler", "type":"Adult", "age":28 }],"travelDate":"17-Dec-2010"} How it works... PHP provides the function json_encode() to create JSON strings from objects and arrays. This function accepts two parameters. First is the value to be encoded and the second parameter includes options that control how certain special characters are encoded. This parameter is optional. In the previous code we created a somewhat complex associative array that contains travel information of two passengers. Passing this array to json_encode() creates a JSON string. There's more... Predefined constants Any of the following constants can be passed as a second parameter to json_encode(). JSON_HEX_TAG: Converts < and > to u003C and u003E JSON_HEX_AMP: Converts &s to u0026 JSON_HEX_APOS: Converts ' to u0027 JSON_HEX_QUOT: Converts " to u0022 JSON_FORCE_OBJECT: Forces the return value in JSON string to be an object instead of an array These constants require PHP version 5.3 or higher.
Read more
  • 0
  • 0
  • 7474
article-image-securing-moodle-instance
Packt
14 Feb 2011
7 min read
Save for later

Securing a Moodle Instance

Packt
14 Feb 2011
7 min read
Moodle Security Moodle is an open source CMS (Course Management System)/LMS (Learning Management System)/VLE (Virtual Learning Environment). Its primary purpose is to enable educational institutions and individuals to create and publish learning content in a coherent and pedagogically valuable manner, so that it can be used for successful knowledge transfer towards students. That sounds harmless enough. Why would anybody want to illegally access an educational platform? There are various motives of computer criminals. In general, they are people committed to the circumvention of computer security. This primarily concerns unauthorized remote computer break-ins via a communication network such as the Internet. Some of the motives could be: Financial: Stealing user and/or course information and selling it to other third-parties Personal: Personal grudge, infantile display of power, desire to alter assigned grades, and so on Weak points Moodle is a web application and as such must be hosted on a computer connected to some kind of network (private or public—Internet / Intranet). This computer must have the following components: Operating System (OS) Web server PHP Database server Moodle Each of these pieces can be used as a point of attack by a malicious user(s) in order to obtain access to the protected information. Therefore, it is our task to make all of them as secure as possible. The main focus will be directed towards our Moodle and PHP configuration. The secure installation of Moodle In this section we follow a secure installation of Moodle. In case you do not already have an installed instance of Moodle, we will show you the quickest way to do that, and at the same time focus on security. If you already have Moodle installed, go to the following section where you will see how to secure an existing installation of Moodle Starting from scratch In order to install Moodle on your server you need to install and configure the web server with support for PHP and the database server. We will not go into the specifics of setting up a particular web server, PHP, and/or database server right now, since it depends on the OS your server has installed. Also we will not explain in detail tasks like creating directories, setting up file permissions, etc as they are OS specific and out of the scope of this article. This section assumes you already know about your OS and have already configured your web server with an empty database. Every installation of Moodle must have: Web server with PHP support Dedicated database Two dedicated directories—one for Moodle and another for platform data We assume that your web server is Apache (Linux) or IIS (Windows), and that you use PHP 5.1.x or later and MySQL 5.0 or later. Installation checklist The following checklist will guide you through the basic installation procedure for Moodle. Download the latest stable version of Moodle from http://download. moodle.org/. (At the time of writing this article it is 1.9.8+). You have two options available on the download page—moodle-weekly-19.tgz or moodle-weekly-19.zip archive. In case you use Linux you can choose either. In case of Windows, ZIP file is the preferred choice. The reason for this is simple. Every Windows server comes, by default, with installed support for managing Zip archives. On the other hand, TGZ is readily available on every Linux distribution. Unpack the compressed file you just downloaded. This will produce a directory with the name moodle which contains all of the platform files. Move that directory to the web-root of your web server. After doing that it is recommended to make all files read-only for safety reasons. Create a directory called moodledata somewhere on the disk. Make sure that it is not in the web-root of your web server since that would incur a serious security breach. Doing that might expose all platform files submitted by course participants and teachers together with the course content to the outside world. Create an empty database (we suggest the name moodle or moodledb). The default database character set must be configured to utf8 and collation set to utf8_general_ci. It is recommended to have a special user for accessing this database with limited permissions. In case of credentials theft, a malicious user could only operate on data from one database, minimizing the potential damage. That database user account will need permissions for creating, altering, and deleting the tables, creating/dropping the indexes and reading/writing the data. Here is what you need to execute in your MySQL console for creating a database and user: CREATE DATABASE moodle CHARSET 'utf8' COLLATION 'utf8_general_ ci'; CREATE USER 'moodle'@'localhost' IDENTIFIED BY 'somepass'; GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER ON loomdb.* TO loom@localhost IDENTIFIED BY 'somepass'; FLUSH PRIVILEGES; Start the installation by opening the http://url to local installation of the moodle (for example http://localhost/moodle) in your browser. Make sure it is a more recent browser with pop ups and JavaScript enabled. We recommend Internet Explorer 8+ or Firefox 3.6+. You will see the following screenshot. On the next screen, we need to specify the web address of the platform and the location of the moodle directory on the disk. Now, we must configure database access. Choose MySQL as database type, localhost as host server, set the name of the database (moodle), database user, and its password (moodle/moodle). You should leave the table prefix as is. Moodle checks the server configuration on this screen and displays the outcome. We can proceed with the installation only if all of the minimal requirements are met. During installation, Moodle generates a configuration file within the moodle directory called config.php. It is important to make this file read-only after installation for security reasons. In case Moodle cannot save config.php it will offer to download or copy content of the file and manually place it in the appropriate location on the server. See the following screenshot: We are now presented with terms of usage and license agreement. To proceed click yes. We can now start the installation itself. During that process Moodle will create all of the tables in the database, session files in the moodledata directory, and load some initial information. Make sure you check Unattended operation at the bottom. That way, the process will be executed without user intervention. After the database setup is finished, we are offered a new screen where we must configure the administrative account. With this user you manage your platform, so be careful about disclosing this information to other users. Field name Description Recommended action Username Defines user name inside the Moodle. By default it is admin. We recommend leaving the default value unchanged. New password Defines user logon password. Must supply valid password. First name Defines name of the admin. Must supply valid name. Surname Defines surname of the admin. Must supply valid name. E-mail address Defines user e-mail address. Must supply valid e-mail. E-mail display Define the visibility of your e-mail address within the platform. We recommend leaving it as is (visible to all). E-mail active Defines whether e-mail is activated or not. Set it to enable. City/Town Defines name of the city where you live. Moodle requires this value. Select Country Name of your country. Set it to your country name. Timezone Sets your time zone so that server can display time calculated for your location in some reports. If not sure what your time zone is, leave it as is.   Preferred language Choose the platform language. By default, Moodle comes only with support for English language. If you want to add more languages visit http://download.moodle.org/ lang16/ and download and install the appropriate files.   After configuring administrative user there is just one more step to complete and that is setting up the site title and short name. In the Full site name field, place the long name you would like to set for your website; it can have multiple words. In the Short name for the site field put one word without spaces which will represent your website. In the Front Page Description field put a longer description (one paragraph) that explains in more detail the purpose of your site. This is optional and does not affect the Moodle functionality at all You have now finished installing Moodle and should see the following screenshot:
Read more
  • 0
  • 0
  • 7472

article-image-implementing-routing-with-react-router-and-graphql-tutorial
Bhagyashree R
19 May 2019
15 min read
Save for later

Implementing routing with React Router and GraphQL [Tutorial]

Bhagyashree R
19 May 2019
15 min read
Routing is essential to most web applications. You cannot cover all of the features of your application in just one page. It would be overloaded, and your user would find it difficult to understand. Sharing links to pictures, profiles, or posts is also very important for a social network such as Graphbook. It is also crucial to split content into different pages, due to search engine optimization (SEO). This article is taken from the book Hands-on Full-Stack Web Development with GraphQL and React by Sebastian Grebe. This book will guide you in implementing applications by using React, Apollo, Node.js, and SQL. By the end of the book, you will be proficient in using GraphQL and React for your full-stack development requirements. To follow along with the examples implemented in this article, you can download the code from the book’s GitHub repository. In this article, we will learn how to do client-side routing in a React application. We will cover the installation of React Router, implement routes, create user profiles with GraphQL backend, and handle manual navigation. Installing React Router We will first start by installing and configuring React Router 4 by running npm: npm install --save react-router-dom From the package name, you might assume that this is not the main package for React. The reason for this is that React Router is a multi-package library. That comes in handy when using the same tool for multiple platforms. The core package is called react-router. There are two further packages. The first one is the react-router-dom package, which we installed in the preceding code, and the second one is the react-router-native package. If at some point, you plan to build a React Native app, you can use the same routing, instead of using the browser's DOM for a real mobile app. The first step that we will take introduces a simple router to get our current application working, including different paths for all of the screens. There is one thing that we have to prepare before continuing. For development, we are using the webpack development server. To get the routing working out of the box, we will add two parameters to the webpack.client.config.js file. The devServer field should look as follows: devServer: { port: 3000, open: true, historyApiFallback: true, }, The historyApiFallback field tells the devServer to serve the index.html file, not only for the root path, http://localhost:3000/ but also when it would typically receive a 404 error. That happens when the path does not match a file or folder that is normal when implementing routing. The output field at the top of the config file must have a publicPath property, as follows: output: { path: path.join(__dirname, buildDirectory), filename: 'bundle.js', publicPath: '/', }, The publicPath property tells webpack to prefix the bundle URL to an absolute path, instead of a relative path. When this property is not included, the browser cannot download the bundle when visiting the sub-directories of our application, as we are implementing client-side routing. Implementing your first route Before implementing the routing, we will clean up the App.js file. Create a Main.js file next to the App.js file in the client folder. Insert the following code: import React, { Component } from 'react'; import Feed from './Feed'; import Chats from './Chats'; import Bar from './components/bar'; import CurrentUserQuery from './components/queries/currentUser'; export default class Main extends Component { render() { return ( <CurrentUserQuery> <Bar changeLoginState={this.props.changeLoginState}/> <Feed /> <Chats /> </CurrentUserQuery> ); }} As you might have noticed, the preceding code is pretty much the same as the logged in condition inside the App.js file. The only change is that the changeLoginState function is taken from the properties, and is not directly a method of the component itself. That is because we split this part out of the App.js and put it into a separate file. This improves reusability for other components that we are going to implement. Now, open and replace the render method of the App component to reflect those changes, as follows: render() { return ( <div> <Helmet> <title>Graphbook - Feed</title> <meta name="description" content="Newsfeed of all your friends on Graphbook" /> </Helmet> <Router loggedIn={this.state.loggedIn} changeLoginState= {this.changeLoginState}/> </div> ) } If you compare the preceding method with the old one, you can see that we have inserted a Router component, instead of directly rendering either the posts feed or the login form. The original components of the App.js file are now in the previously created Main.js file. Here, we pass the loggedIn state variable and the changeLoginState function to the Router component. Remove the dependencies at the top, such as the Chats and Feed components, because we won't use them any more thanks to the new Main component. Add the following line to the dependencies of our App.js file: import Router from './router'; To get this code working, we have to implement our custom Router component first. Generally, it is easy to get the routing running with React Router, and you are not required to separate the routing functionality into a separate file, but, that makes it more readable. To do this, create a new router.js file in the client folder, next to the App.js file, with the following content: import React, { Component } from 'react'; import LoginRegisterForm from './components/loginregister'; import Main from './Main'; import { BrowserRouter as Router, Route, Redirect, Switch } from 'react-router-dom'; export default class Routing extends Component { render() { return ( <Router> <Switch> <Route path="/app" component={() => <Main changeLoginState= {this.props.changeLoginState}/>}/> </Switch> </Router> ) }} At the top, we import all of the dependencies. They include the new Main component and the react-router package. The problem with the preceding code is that we are only listening for one route, which is /app. If you are not logged in, there will be many errors that are not covered. The best thing to do would be to redirect the user to the root path, where they can log in. Advanced routing with React Router The primary goal of this article is to build a profile page, similar to Facebook, for your users. We need a separate page to show all of the content that a single user has entered or created. Parameters in routes We have prepared most of the work required to add a new user route. Open up the router.js file again. Add the new route, as follows: <PrivateRoute path="/user/:username" component={props => <User {...props} changeLoginState={this.props.changeLoginState}/>} loggedIn={this.props.loggedIn}/> Those are all of the changes that we need to accept parameterized paths in React Router. We read out the value inside of the new user page component. Before implementing it, we import the dependency at the top of router.js to get the preceding route working: import User from './User'; Create the preceding User.js file next to the Main.js file. Like the Main component, we are collecting all of the components that we render on this page. You should stay with this layout, as you can directly see which main parts each page consists of. The User.js file should look as follows: import React, { Component } from 'react'; import UserProfile from './components/user'; import Chats from './Chats'; import Bar from './components/bar'; import CurrentUserQuery from './components/queries/currentUser'; export default class User extends Component { render() { return ( <CurrentUserQuery> <Bar changeLoginState={this.props.changeLoginState}/> <UserProfile username={this.props.match.params.username}/> <Chats /> </CurrentUserQuery> ); }} We use the CurrentUserQuery component as a wrapper for the Bar component and the Chats component. If a user visits the profile of a friend, they see the common application bar at the top. They can access their chats on the right-hand side, like in Facebook. We removed the Feed component and replaced it with a new UserProfile component. Importantly, the UserProfile receives the username property. Its value is taken from the properties of the User component. These properties were passed over by React Router. If you have a parameter, such as a username, in the routing path, the value is stored in the match.params.username property of the child component. The match object generally contains all matching information of React Router. From this point on, you can implement any custom logic that you want with this value. We will now continue with implementing the profile page. Follow these steps to build the user's profile page: Create a new folder, called user, inside the components folder. Create a new file, called index.js, inside the user folder. Import the dependencies at the top of the file, as follows: import React, { Component } from 'react'; import PostsQuery from '../queries/postsFeed'; import FeedList from '../post/feedlist'; import UserHeader from './header'; import UserQuery from '../queries/userQuery'; The first three lines should look familiar. The last two imported files, however, do not exist at the moment, but we are going to change that shortly. The first new file is UserHeader, which takes care of rendering the avatar image, the name, and information about the user. Logically, we request the data that we will display in this header through a new Apollo query, called UserQuery. Insert the code for the UserProfile component that we are building at the moment beneath the dependencies, as follows: export default class UserProfile extends Component { render() { const query_variables = { page: 0, limit: 10, username: this.props.username }; return ( <div className="user"> <div className="inner"> <UserQuery variables={{username: this.props.username}}> <UserHeader/> </UserQuery> </div> <div className="container"> <PostsQuery variables={query_variables}> <FeedList/> </PostsQuery> </div> </div> ) } } The UserProfile class is not complex. We are running two Apollo queries simultaneously. Both have the variables property set. The PostQuery receives the regular pagination fields, page and limit, but also the username, which initially came from React Router. This property is also handed over to the UserQuery, inside of a variables object. We should now edit and create the Apollo queries, before programming the profile header component. Open the postsFeed.js file from the queries folder. To use the username as input to the GraphQL query we first have to change the query string from the GET_POSTS variable. Change the first two lines to match the following code: query postsFeed($page: Int, $limit: Int, $username: String) { postsFeed(page: $page, limit: $limit, username: $username) { Add a new line to the getVariables method, above the return statement: if(typeof variables.username !== typeof undefined) { query_variables.username = variables.username; } If the custom query component receives a username property, it is included in the GraphQL request. It is used to filter posts by the specific user that we are viewing. Create a new userQuery.js file in the queries folder to create the missing query class. Import all of the dependencies and parse the new query schema with graphl-tag, as follows: import React, { Component } from 'react'; import { Query } from 'react-apollo'; import Loading from '../loading'; import Error from '../error'; import gql from 'graphql-tag'; const GET_USER = gql` query user($username: String!) { user(username: $username) { id email username avatar } }`; The preceding query is nearly the same as the currentUser query. We are going to implement the corresponding user query later, in our GraphQL API. The component itself is as simple as the ones that we created before. Insert the following code: export default class UserQuery extends Component { getVariables() { const { variables } = this.props; var query_variables = {}; if(typeof variables.username !== typeof undefined) { query_variables.username = variables.username; } return query_variables; } render() { const { children } = this.props; const variables = this.getVariables(); return( <Query query={GET_USER} variables={variables}> {({ loading, error, data }) => { if (loading) return <Loading />; if (error) return <Error><p>{error.message}</p></Error>; const { user } = data; return React.Children.map(children, function(child){ return React.cloneElement(child, { user }); }) }} </Query> ) } } We set the query property and the parameters that are collected by the getVariables method to the GraphQL Query component. The rest is the same as any other query component that we have written. All child components receive a new property, called user, which holds all the information about the user, such as their name, their email, and their avatar image. The last step is to implement the UserProfileHeader component. This component renders the user property, with all its values. It is just simple HTML markup. Copy the following code into the header.js file, in the user folder: import React, { Component } from 'react';export default class UserProfileHeader extends Component { render() { const { avatar, email, username } = this.props.user; return ( <div className="profileHeader"> <div className="avatar"> <img src={avatar}/> </div> <div className="information"> <p> {username} </p> <p> {email} </p> <p>You can provide further information here and build your really personal header component for your users.</p> </div> </div> ) }} We have finished the new front end components, but the UserProfile component is still not working. The queries that we are using here either do not accept the username parameter or have not yet been implemented. Querying the user profile With the new profile page, we have to update our back end accordingly. Let's take a look at what needs to be done, as follows: We have to add the username parameter to the schema of the postsFeed query and adjust the resolver function. We have to create the schema and the resolver function for the new UserQuery component. We will begin with the postsFeed query. Edit the postsFeed query in the RootQuery type of the schema.js file to match the following code: postsFeed(page: Int, limit: Int, username: String): PostFeed @auth Here, I have added the username as an optional parameter. Now, head over to the resolvers.js file, and take a look at the corresponding resolver function. Replace the signature of the function to extract the username from the variables, as follows: postsFeed(root, { page, limit, username }, context) { To make use of the new parameter, add the following lines of code above the return statement: if(typeof username !== typeof undefined) { query.include = [{model: User}]; query.where = { '$User.username$': username }; } In the preceding code, we fill the include field of the query object with the Sequelize model that we want to join. This allows us to filter the associated Chats model in the next step. Then, we create a normal where object, in which we write the filter condition. If you want to filter the posts by an associated table of users, you can wrap the model and field names that you want to filter by with dollar signs. In our case, we wrap User.username with dollar signs, which tells Sequelize to query the User model's table and filter by the value of the username column. No adjustments are required for the pagination part. The GraphQL query is now ready. The great thing about the small changes that we have made is that we have just one API function that accepts several parameters, either to display posts on a single user profile, or to display a list of posts like a news feed. Let's move on and implement the new user query. Add the following line to the RootQuery in your GraphQL schema: user(username: String!): User @auth This query only accepts a username, but this time it is a required parameter in the new query. Otherwise, the query would make no sense, since we only use it when visiting a user's profile through their username. In the resolvers.js file, we will now implement the resolver function using Sequelize: user(root, { username }, context) { return User.findOne({ where: { username: username } }); }, In the preceding code, we use the findOne method of the User model by Sequelize, and search for exactly one user with the username that we provided in the parameter. We also want to display the email of the user on the user's profile page. Add the email as a valid field on the User type in your GraphQL schema with the following line of code: email: String With this step, our back end code and the user page are ready. This article walked you through the installation process of React Router and how to implement a route in React. Then we moved on to more advanced stuff by implementing a user profile, similar to Facebook, with a GraphQL backend. If you found this post useful, do check out the book, Hands-on Full-Stack Web Development with GraphQL and React. This book teaches you how to build scalable full-stack applications while learning to solve complex problems with GraphQL. How to build a Relay React App [Tutorial] React vs. Vue: JavaScript framework wars Working with the Vue-router plugin for SPAs
Read more
  • 0
  • 0
  • 7463

article-image-minecraft-java-team-are-open-sourcing-some-of-minecrafts-code-as-libraries
Sugandha Lahoti
08 Oct 2018
2 min read
Save for later

Minecraft Java team are open sourcing some of Minecraft's code as libraries

Sugandha Lahoti
08 Oct 2018
2 min read
Stockholm's Minecraft Java team are open sourcing some of Minecraft's code as libraries for game developers. Developers can now use them to improve their Minecraft mods, use them for their own projects, or help improve pieces of the Minecraft Java engine. The team will open up different libraries gradually. These libraries are open source and MIT licensed. For now, they have open sourced two libraries Brigadier and DataFixerUpper. Brigadier The first library, Brigadier takes random strings of text entered into Minecraft and turns into an actual function that the game will perform. Basically, if you enter in the game something like /give Dinnerbone sticks, it goes internally into Brigadier and breaks it down into pieces. Then it tries to figure out what the developer is trying to do with this random piece of text. Nathan Adams, a Java developer hopes that giving the Minecraft community access to Brigadier can make it “extremely user-friendly one day.” Brigadier has been available for a week now. It has already seen improvements in the code and the readme doc. DataFixerUpper Another important library of the Minecraft game engine, the DataFixerUpper is also being open sourced. When a developer adds a new feature into Minecraft, they have to change the way level data and save files are stored. DataFixerUpper turns these data formats to what the game should currently be using now. Also in consideration for open sourcing is the Blaze3D library, which is a complete rewrite of the render engine for Minecraft 1.14. You can check out the announcement on the Minecraft website. You can also download Brigadier and DataFixerUpper. Minecraft is serious about global warming, adds a new (spigot) plugin to allow changes in climate mechanics. Learning with Minecraft Mods A Brief History of Minecraft Modding
Read more
  • 0
  • 0
  • 7451
article-image-remote-sensing-and-histogram
Packt
04 Jan 2016
10 min read
Save for later

Remote Sensing and Histogram

Packt
04 Jan 2016
10 min read
In this article by Joel Lawhead, the author of Learning GeoSpatial Analysis with Python - Second Edition, we will discuss remote sensing. This field grows more exciting every day as more satellites are launched and the distribution of data becomes easier. The high availability of satellite and aerial images as well as the interesting new types of sensors that are being launched each year is changing the role that remote sensing plays in understanding our world. In this field, Python is quite capable. In remote sensing, we step through each pixel in an image and perform a type of query or mathematical process. An image can be thought of as a large numerical array and in remote sensing, these arrays can be as large as tens of megabytes to several gigabytes. While Python is fast, only C-based libraries can provide the speed that is needed to loop through the arrays at a tolerable speed. (For more resources related to this topic, see here.) In this article, whenever possible, we’ll use Python Imaging Library (PIL) for image processing and NumPy, which provides multidimensional array mathematics. While written in C for speed, these libraries are designed for Python and provide Python’s API. In this article, we’ll start with basic image manipulation and build on each exercise all the way to automatic change detection. Here are the topics that we’ll cover: Swapping image bands Creating image histograms Swapping image bands Our eyes can only see colors in the visible spectrum as combinations of red, green, and blue (RGB). Airborne and spaceborne sensors can collect wavelengths of the energy that is outside the visible spectrum. In order to view this data, we move images representing different wavelengths of light reflectance in and out of the RGB channels in order to make color images. These images often end up as bizarre and alien color combinations that can make visual analysis difficult. An example of a typical satellite image is seen in the following Landsat 7 satellite scene near the NASA's Stennis Space Center in Mississippi along the Gulf of Mexico, which is a leading space center for remote sensing and geospatial analysis in general: Most of the vegetation appears red and water appears almost black. This image is one type of false color image, meaning that the color of the image is not based on the RGB light. However, we can change the order of the bands or swap certain bands in order to create another type of false color image that looks more like the world we are used to seeing. In order to do so, you first need to download this image as a ZIP file from the following: http://git.io/vqs41 We need to install the GDAL library with Python bindings. The Geospatial Data Abstraction Library(GDAL)includes a module called gdalnumeric that loads and saves remotely sensed images to and from NumPy arrays for easy manipulation. GDAL itself is a data access library and does not provide much in the name of processing. Therefore, in this article, we will rely heavily on NumPy to actually change images. In this example, we’ll load the image in a NumPy array using gdalnumeric and then, we’ll immediately save it in a new .tiff file. However, upon saving, we’ll use NumPy’s advanced array slicing feature to change the order of the bands. Images in NumPy are multidimensional arrays in the order of band, height, and width. Therefore, an image with three bands will be an array of length three, containing an array for each band, the height, and width of the image. It’s important to note that NumPy references the array locations as y,x (row, column) instead of the usual column and row format that we work with in spreadsheets and other software: from osgeo import gdalnumeric src = "FalseColor.tif" arr = gdalnumeric.LoadFile(src) gdalnumeric.SaveArray(arr[[1, 0, 2], :], "swap.tif",format="GTiff", prototype=src) Also in the SaveArray() method, the last argument is called prototype. This argument let’s you specify another image for GDAL from which we can copy spatial reference information and some other image parameters. Without this argument, we’d end up with an image without georeferencing information, which can not be used in a geographical information system (GIS). In this case, we specified our input image file name as the images are identical, except for the band order. The result of this example produces the swap.tif image, which is a much more visually appealing image with green vegetation and blue water: There’s only one problem with this image. It’s a little dark and difficult to see. Let’s see if we can figure out why. Creating histograms A histogram shows the statistical frequency of data distribution in a dataset. In the case of remote sensing, the dataset is an image, the data distribution is the frequency of the pixels in the range of 0 to 255, which is the range of the 8-byte numbers used to store image information on computers. In an RGB image, color is represented as a three-digit tuple with (0,0,0) being black and (255,255,255) being white. We can graph the histogram of an image with the frequency of each value along the y axis and the range of 255 possible pixel values along the x axis. We can use the Turtle graphics engine that is included with Python to create a simple GIS. We can also use it to easily graph histograms. Histograms are usually a one-off product that makes a quick script, like this example, great. Also, histograms are typically displayed as a bar graph with the width of the bars representing the size of the grouped data bins. However, in an image, each bin is only one value, so we’ll create a line graph. We’ll use the histogram function in this example and create a red, green, and blue line for each band. The graphing portion of this example also defaults to scaling the y axis values to the maximum RGB frequency that is found in the image. Technically, the y axis represents the maximum frequency, that is, the number of pixels in the image, which would be the case if the image was of one color. We’ll use the turtle module again; however, this example could be easily converted to any graphical output module. However, this format makes the distribution harder to see. Let’s take a look at our swap.tif image: from osgeo import gdalnumeric import turtle as t def histogram(a, bins=list(range(0, 256))): fa = a.flat n = gdalnumeric.numpy.searchsorted(gdalnumeric.numpy.sort(fa), bins) n = gdalnumeric.numpy.concatenate([n, [len(fa)]]) hist = n[1:]-n[:-1] return hist defdraw_histogram(hist, scale=True): t.color("black") axes = ((-355, -200), (355, -200), (-355, -200), (-355, 250)) t.up() for p in axes: t.goto(p) t.down() t.up() t.goto(0, -250) t.write("VALUE", font=("Arial, ", 12, "bold")) t.up() t.goto(-400, 280) t.write("FREQUENCY", font=("Arial, ", 12, "bold")) x = -355 y = -200 t.up() for i in range(1, 11): x = x+65 t.goto(x, y) t.down() t.goto(x, y-10) t.up() t.goto(x, y-25) t.write("{}".format((i*25)), align="center") x = -355 y = -200 t.up() pixels = sum(hist[0]) if scale: max = 0 for h in hist: hmax = h.max() if hmax> max: max = hmax pixels = max label = pixels/10 for i in range(1, 11): y = y+45 t.goto(x, y) t.down() t.goto(x-10, y) t.up() t.goto(x-15, y-6) t.write("{}" .format((i*label)), align="right") x_ratio = 709.0 / 256 y_ratio = 450.0 / pixels colors = ["red", "green", "blue"] for j in range(len(hist)): h = hist[j] x = -354 y = -199 t.up() t.goto(x, y) t.down() t.color(colors[j]) for i in range(256): x = i * x_ratio y = h[i] * y_ratio x = x - (709/2) y = y + -199 t.goto((x, y)) im = "swap.tif" histograms = [] arr = gdalnumeric.LoadFile(im) for b in arr: histograms.append(histogram(b)) draw_histogram(histograms) t.pen(shown=False) t.done() Here's what the histogram for swap.tif looks similar to after running the example: As you can see, all the three bands are grouped closely towards the left-hand side of the graph and all have values that are less than 125. As these values approach zero, the image becomes darker, which is not surprising. Just for fun, let’s run the script again and when we call the draw_histogram() function, we’ll add the scale=False option to get an idea of the size of the image and provide an absolute scale. Therefore, we take the following line: draw_histogram(histograms) Change it to the following: draw_histogram(histograms, scale=False) This change will produce the following histogram graph: As you can see, it’s harder to see the details of the value distribution. However, this absolute scale approach is useful if you are comparing multiple histograms from different products that are produced from the same source image. Now that we understand the basics of looking at an image statistically using histograms, how do we make our image brighter? Performing a histogram stretch A histogram stretch operation does exactly what the name suggests. It distributes the pixel values across the whole scale. By doing so, we have more values at the higher-intensity level and the image becomes brighter. Therefore, in this example, we’ll use our histogram function; however, we’ll add another function called stretch() that takes an image array, creates the histogram, and then spreads out the range of values for each band. We’ll run these functions on swap.tif and save the result in an image called stretched.tif: import gdalnumeric import operator from functools import reduce def histogram(a, bins=list(range(0, 256))): fa = a.flat n = gdalnumeric.numpy.searchsorted(gdalnumeric.numpy.sort(fa), bins) n = gdalnumeric.numpy.concatenate([n, [len(fa)]]) hist = n[1:]-n[:-1] return hist def stretch(a): hist = histogram(a) lut = [] for b in range(0, len(hist), 256): step = reduce(operator.add, hist[b:b+256]) / 255 n = 0 for i in range(256): lut.append(n / step) n = n + hist[i+b] gdalnumeric.numpy.take(lut, a, out=a) return a src = "swap.tif" arr = gdalnumeric.LoadFile(src) stretched = stretch(arr) gdalnumeric.SaveArray(arr, "stretched.tif", format="GTiff", prototype=src) The stretch algorithm will produce the following image. Look how much brighter and visually appealing it is: We can run our turtle graphics histogram script on stretched.tif by changing the filename in the im variable to stretched.tif: im = "stretched.tif" This run will give us the following histogram: As you can see, all the three bands are distributed evenly now. Their relative distribution to each other is the same; however, in the image, they are now spread across the spectrum. Summary In this article, we covered the foundations of remote sensing including band swapping and histograms. The authors of GDAL have a set of Python examples, covering some advanced topics that may be of interest, available at https://svn.osgeo.org/gdal/trunk/gdal/swig/python/samples/. Resources for Article: Further resources on this subject: Python Libraries for Geospatial Development[article] Python Libraries[article] Learning R for Geospatial Analysis [article]
Read more
  • 0
  • 0
  • 7448

article-image-introducing-building-blocks-unity-scripts
Packt
11 Oct 2013
15 min read
Save for later

Introducing the Building Blocks for Unity Scripts

Packt
11 Oct 2013
15 min read
(For more resources related to this topic, see here.) Using the term method instead of function You are constantly going to see the words function and method used everywhere as you learn Unity. The words function and method truly mean the same thing in Unity. They do the same thing. Since you are studying C#, and C# is an Object-Oriented Programming (OOP) language, I will use the word "method" throughout this article, just to be consistent with C# guidelines. It makes sense to learn the correct terminology for C#. Also, UnityScript and Boo are OOP languages. The authors of the Scripting Reference probably should have used the word method instead of function in all documentation. From now on I'm going to use the words method or methods in this article. When I refer to the functions shown in the Scripting Reference , I'm going to use the word method instead, just to be consistent throughout this article. Understanding what a variable does in a script What is a variable? Technically, it's a tiny section of your computer's memory that will hold any information you put there. While a game runs, it keeps track of where the information is stored, the value kept there, and the type of the value. However, for this article, all you need to know is how a variable works in a script. It's very simple. What's usually in a mailbox, besides air? Well, usually there's nothing but occasionally there is something in it. Sometimes there's money (a paycheck), bills, a picture from aunt Mabel, a spider, and so on. The point is what's in a mailbox can vary. Therefore, let's call each mailbox a variable instead. Naming a variable Using the picture of the country mailboxes, if I asked you to see what is in the mailbox, the first thing you'd ask is which one? If I said in the Smith mailbox, or the brown mailbox, or the round mailbox, you'd know exactly which mailbox to open to retrieve what is inside. Similarly, in scripts, you have to name your variables with a unique name. Then I can ask you what's in the variable named myNumber, or whatever cool name you might use. A variable name is just a substitute for a value As you write a script and make a variable, you are simply creating a placeholder or a substitute for the actual information you want to use. Look at the following simple math equation: 2 + 9 = 11 Simple enough. Now try the following equation: 11 + myNumber = ??? There is no answer to this yet. You can't add a number and a word. Going back to the mailbox analogy, write the number 9 on a piece of paper. Put it in the mailbox named myNumber. Now you can solve the equation. What's the value in myNumber? The value is 9. So now the equation looks normal: 11 + 9 = 20 The myNumber variable is nothing more than a named placeholder to store some data (information). So anywhere you would like the number 9 to appear in your script, just write myNumber, and the number 9 will be substituted. Although this example might seem silly at first, variables can store all kinds of data that is much more complex than a simple number. This is just a simple example to show you how a variable works. Time for action – creating a variable and seeing how it works Let's see how this actually works in our script. Don't be concerned about the details of how to write this, just make sure your script is the same as the script shown in the next screenshot. In the Unity Project panel, double-click on LearningScript. In MonoDevelop, write the lines 6, 11, and 13 from the next screenshot. Save the file. To make this script work, it has to be attached to a GameObject. Currently, in our State Machine project we only have one GameObject, the Main Camera. This will do nicely since this script doesn't affect the Main Camera in any way. The script simply runs by virtue of it being attached to a GameObject. Drag LearningScript onto the Main Camera. Select Main Camera so that it appears in the Inspector panel. Verify whether LearningScript is attached. Open the Unity Console panel to view the output of the script. Click on Play. The preceding steps are shown in the following screenshot: What just happened? In the following Console panel is the result of our equations. As you can see, the equation on line 13 worked by substituting the number 9 for the myNumber variable: Time for action – changing the number 9 to a different number Since myNumber is a variable, the value it stores can vary. If we change what is stored in it, the answer to the equation will change too. Follow the ensuing steps: Stop the game and change 9 to 19. Notice that when you restart the game, the answer will be 30. What just happened? You learned that a variable works by simple process of substitution. There's nothing more to it than that. We didn't get into the details of the wording used to create myNumber, or the types of variables you can create, but that wasn't the intent. This was just to show you how a variable works. It just holds data so you can use that data elsewhere in your script. Have a go hero – changing the value of myNumber In the Inspector panel, try changing the value of myNumber to some other value, even a negative value. Notice the change in answer in the Console. Using a method in a script Methods are where the action is and where the tasks are performed. Great, that's really nice to know but what is a method? What is a method? When we write a script, we are making lines of code that the computer is going to execute, one line at a time. As we write our code, there will be things we want our game to execute more than once. For example, we can write a code that adds two numbers. Suppose our game needs to add the two numbers together a hundred different times during the game. So you say, "Wow, I have to write the same code a hundred times that adds two numbers together. There has to be a better way." Let a method take away your typing pain. You just have to write the code to add two numbers once, and then give this chunk of code a name, such as AddTwoNumbers(). Now, every time our game needs to add two numbers, don't write the code over and over, just call the AddTwoNumbers() method. Time for action – learning how a method works We're going to edit LearningScript again. In the following screenshot, there are a few lines of code that look strange. We are not going to get into the details of what they mean in this article.Getting into the Details of Methods. Right now, I am just showing you a method's basic structure and how it works: In MonoDevelop, select LearningScript for editing. Edit the file so that it looks exactly like the following screenshot. Save the file. What's in this script file? In the previous screenshot, lines 6 and 7 will look familiar to you; they are variables just as you learned in the previous section. There are two of them this time. These variables store the numbers that are going to be added. Line 16 may look very strange to you. Don't concern yourself right now with how this works. Just know that it's a line of code that lets the script know when the Return/Enter key is pressed. Press the Return/Enter key when you want to add the two numbers together. Line 17 is where the AddTwoNumbers() method gets called into action. In fact, that's exactly how to describe it. This line of code calls the method. Lines 20, 21, 22, and 23 make up the AddTwoNumbers() method. Don't be concerned about the code details yet. I just want you to understand how calling a method works. Method names are substitutes too You learned that a variable is a substitute for the value it actually contains. Well, a method is no different. Take a look at line 20 from the previous screenshot: void AddTwoNumbers () The AddTwoNumbers() is the name of the method. Like a variable, AddTwoNumbers() is nothing more than a named placeholder in the memory, but this time it stores some lines of code instead. So anywhere we would like to use the code of this method in our script, just write AddTwoNumbers(), and the code will be substituted. Line 21 has an opening curly-brace and line 23 has a closing curly-brace. Everything between the two curly-braces is the code that is executed when this method is called in our script. Look at line 17 from the previous screenshot: AddTwoNumbers(); The method name AddTwoNumbers() is called. This means that the code between the curly-braces is executed. It's like having all of the code of a method right there on line 17. Of course, this AddTwoNumbers() method only has one line of code to execute, but a method could have many lines of code. Line 22 is the action part of this method, the part between the curly-braces. This line of code is adding the two variables together and displaying the answer to the Unity Console. Then, follow the ensuing steps: Go back to Unity and have the Console panel showing. Now click on Play. What just happened? Oh no! Nothing happened! Actually, as you sit there looking at the blank Console panel, the script is running perfectly, just as we programmed it. Line 16 in the script is waiting for you to press the Return/Enter key. Press it now. And there you go! The following screenshot shows you the result of adding two variables together that contain the numbers 2 and 9: Line 16 waited for you to press the Return/Enter key. When you do this, line 17 executes which calls the AddTwoNumbers() method. This allows the code block of the method, line 23, to add the the values stored in the variables number1 and number2. Have a go hero – changing the output of the method While Unity is in the Play mode, select the Main Camera so its Components show in the Inspector. In the Inspector panel, locate Learning Script and its two variables. Change the values, currently 2 and 9, to different values. Make sure to click your mouse in the Game panel so it has focus, then press the Return/Enter key again. You will see the result of the new addition in the Console. You just learned how a method works to allow a specific block of code to to be called to perform a task. We didn't get into any of the wording details of methods here, this was just to show you fundamentally how they work. Introducing the class The class plays a major role in Unity. In fact, what Unity does with a class a little piece of magic when Unity creates Components. You just learned about variables and methods. These two items are the building blocks used to build Unity scripts. The term script is used everywhere in discussions and documents. Look it up in the dictionary and it can be generally described as written text. Sure enough, that's what we have. However, since we aren't just writing a screenplay or passing a note to someone, we need to learn the actual terms used in programming. Unity calls the code it creates a C# script. However, people like me have to teach you some basic programming skills and tell you that a script is really a class. In the previous section about methods, we created a class (script) called LearningScript. It contained a couple of variables and a method. The main concept or idea of a class is that it's a container of data, stored in variables, and methods that process that data in some fashion. Because I don't have to constantly write class (script), I will be using the word script most of the time. However, I will also be using class when getting more specific with C#. Just remember that a script is a class that is attached to a GameObject. The State Machine classes will not be attached to any GameObjects, so I won't be calling them scripts. By using a little Unity magic, a script becomes a Component While working in Unity, we wear the following two hats: A Game-Creator hat A Scripting (programmer) hat When we first wear our Game-Creator hat, we will be developing our Scene, selecting GameObjects, and viewing Components; just about anything except writing our scripts. When we put our Scripting hat on, our terminology changes as follows: We're writing code in scripts using MonoDevelop We're working with variables and methods The magic happens when you put your Game-Creator hat back on and attach your script to a GameObject. Wave the magic wand — ZAP — the script file is now called a Component, and the public variables of the script are now the properties you can see and change in the Inspector panel. A more technical look at the magic A script is like a blueprint or a written description. In other words, it's just a single file in a folder on our hard drive. We can see it right there in the Projects panel. It can't do anything just sitting there. When we tell Unity to attach it to a GameObject, we haven't created another copy of the file, all we've done is tell Unity we want the behaviors described in our script to be a Component of the GameObject. When we click on the Play button, Unity loads the GameObject into the computer's memory. Since the script is attached to a GameObject, Unity also has to make a place in the computer's memory to store a Component as part of the GameObject. The Component has the capabilities specified in the script (blueprint) we created. Even more Unity magic There's some more magic you need to be aware of. The scripts inherit from MonoBehaviour. For beginners to Unity, studying C# inheritance isn't a subject you need to learn in any great detail, but you do need to know that each Unity script uses inheritance. We see the code in every script that will be attached to a GameObject. In LearningScript, the code is on line 4: public class LearningScript : MonoBehaviour The colon and the last word of that code means that the LearningScript class is inheriting behaviors from the MonoBehaviour class. This simply means that the MonoBehaviour class is making few of its variables and methods available to the LearningScript class. It's no coincidence that the variables and methods inherited look just like some of the code we saw in the Unity Scripting Reference. The following are the two inherited behaviors in the LearningScript: Line 9:: void Start () Line 14: void Update () The magic is that you don't have to call these methods, Unity calls them automatically. So the code you place in these methods gets executed automatically. Have a go hero – finding Start and Update in the Scripting Reference Try a search on the Scripting Reference for Start and Update to learn when each method is called by Unity and how often. Also search for MonoBehaviour. This will show you that since our script inherits from MonoBehaviour, we are able to use the Start() and Update() methods. Components communicating using the Dot Syntax Our script has variables to hold data, and our script has methods to allow tasks to be performed. I now want to introduce the concept of communicating with other GameObjects and the Components they contain. Communication between one GameObject's Components and another GameObject's Components using Dot Syntax is a vital part of scripting. It's what makes interaction possible. We need to communicate with other Components or GameObjects to be able to use the variables and methods in other Components. What's with the dots? When you look at the code written by others, you'll see words with periods separating them. What the heck is that? It looks complicated, doesn't it. The following is an example from the Unity documentation: transform.position.x Don't concern yourself with what the preceding code means as that comes later, I just want you to see the dots. That's called the Dot Syntax. The following is another example. It's the fictitious address of my house: USA.Vermont.Essex.22MyStreet Looks funny, doesn't it? That's because I used the syntax (grammar) of C# instead of the post office. However, I'll bet if you look closely, you can easily figure out how to find my house. Summary This article introduced you to the basic concepts of variables, methods, and Dot Syntax. These building blocks are used to create scripts and classes. Understanding how these building blocks work is critical so you don't feel you're not getting it. We discovered that a variable name is a substitute for the value it stores; a method name is a substitute for a block of code; when a script or class is attached to a GameObject, it becomes a Component. The Dot Syntax is just like an address to locate GameObjects and Components. With these concepts under your belt, we can proceed to learn the details of the sentence structure, the grammar, and the syntax used to work with variables, methods, and the Dot Syntax. Resources for Article: Further resources on this subject: Debugging Multithreaded Applications as Singlethreaded in C# [Article] Simplifying Parallelism Complexity in C# [Article] Unity Game Development: Welcome to the 3D world [Article]
Read more
  • 0
  • 0
  • 7439