Tech Guides

When you begin using Python after using other languages, it's easy to bring a lot of idioms with you. Though they may work, they are not the best, most beautiful, or fastest ways to get things done with Python—they're not pythonic. I've put together some tips on basic things that can provide big performance improvements, and I hope they'll serve as a starting point for you as you develop with Python. Use comprehensions Comprehensions are great. Python knows how to make lists, tuples, sets, and dicts from single statements, so you don't need to declare, initialize, and append things to your sequences as you do in Java. It helps not only in readability but also on performance; if you delegate something to the interpreter, it will make it faster. def do_something_with(value): return value * 2 # this is an anti-pattern my_list = [] for value in range(10): my_list.append(do_something_with(value)) # this is beautiful and faster my_list = [do_something_with(value) for value in range(10)] # and you can even plug some validation def some_validation(value): return value % 2 my_list = [do_something_with(value) for value in range(10) if some_validation(value)] my_list [2, 6, 10, 14, 18] And it looks the same for other types. You just need to change the appropriate surrounding symbols to get what you want. my_tuple = tuple(do_something_with(value) for value in range(10)) my_tuple (0, 2, 4, 6, 8, 10, 12, 14, 16, 18) my_set = {do_something_with(value) for value in range(10)} my_set {0, 2, 4, 6, 8, 10, 12, 14, 16, 18} my_dict = {value: do_something_with(value) for value in range(10)} my_dict {0: 0, 1: 2, 2: 4, 3: 6, 4: 8, 5: 10, 6: 12, 7: 14, 8: 16, 9: 18} Use Generators Generators are objects that generate sequences one item at a time. This provides a great gain in performance when working with large data, because it won't generate the whole sequence unless needed, and it’s also a memory saver. A simple way to use generators is very similar to the comprehensions we saw above, but you encase the sentence with () instead of [] for example: my_generator = (do_something_with(value) for value in range(10)) my_generator <generator object <genexpr> at 0x7f0d31c207e0> The range function itself returns a generator (unless you're using legacy Python 2, in which case you need to use xrange). Once you have a generator, call next to iterate over its items, or use it as a parameter to a sequence constructor if you really need all the values: next(my_generator) 0 next(my_generator) 2 To create your own generators, use the yield keyword inside a loop instead of a regular return at the end of a function or method. Each time you call next on it, your code will run until it reaches a yield statement, and it saves the state for the next time you ask for a value. # this is a generator that infinitely returns a sequence of numbers, adding 1 to the previous def my_generator_creator(start_value): while True: yield start_value start_value += 1 my_integer_generator = my_generator_creator(0) my_integer_generator <generator object my_generator_creator at 0x7f0d31c20708> next(my_integer_generator) 0 next(my_integer_generator) 1 The benefits of generators in this case are obvious—you would never end up generating numbers if you were to create the whole sequence before using it. A great use of this, for example, is for reading a file stream. Use sets for membership checks It's wonderful how we can use the in keyword to check for membership on any type of sequence. But sets are special. They're of a mapping kind, an unordered set of values where an item's positions are calculated rather than searched. When you search for a value inside a list, the interpreter searches the entire list to see whether the value is there. If you are lucky, the value is the first of the sequence; on the other hand, it could even be the last in a long list. When working with sets, the membership check always takes the same time because the positions are calculated and the interpreter knows where to search for the value. If you're using long sets or loops, the performance gain is sensible. You can create a set from any iterable object as long as the values are hashable. my_list_of_fruits = ['apple', 'banana', 'coconut', 'damascus'] my_set_of_fruits = set(my_list_of_fruits) my_set_of_fruits {'apple', 'banana', 'coconut', 'damascus'} 'apple' in my_set_of_fruits True 'watermelon' in my_set_of_fruits False Deal with strings the right way You've probably done or read something like this before: # this is an anti-pattern "some string, " + "some other string, " + "and yet another one" 'some string, some other string, and yet another one' It might look easy and fast to write, but it's terrible for your performance. str objects are immutable, so each time you add strings, trying to append them, you're actually creating new strings. There are a handful of methods to deal with strings in a faster and optimal way. To join strings, use the join method on a separator with a sequence of strings. The separator can be an empty string if you just want to concatenate. # join a sequence of strings, based on the separator you want ", ".join(["some string", "some other string", "and yet another one"]) 'some string, some other string, and yet another one' ''.join(["just", "concatenate"]) 'justconcatenate' To merge strings, for example, to insert information in templates, we have a classical way; it resembles the C language: # the classical way "hello %s, my name is %s" % ("everyone", "Gabriel") 'hello everyone, my name is Gabriel' And then there is the modern way, with the format method. It is quite flexible: # formatting with sequencial strings "hello {}, my name is {}".format("everyone", "Gabriel") 'hello everyone, my name is Gabriel' # formatting with indexed strings "hello {1}, my name is {2} and we all love {0}".format("Python", "everyone", "Gabriel") 'hello everyone, my name is Gabriel and we all love Python' Avoid intermediate outputs Every programmer in this world has used print statements for debugging or progress checking purposes at least once. If you don't know pdb for debugging yet, you should check it out immediately. But I'll agree that it's really easy to write print statements inside your loops to keep track of where your program is. I'll just tell you to avoid them because they're synchronous and will significantly raise the execution time. You can think of alternative ways to check progress, such as watching via the filesystem the files that you have to generate anyway. Asynchronous programming is a huge topic that you should take a look at if you're dealing with a lot of I/O operations. Cache the most requested results Caching is one of the greatest performance tunning tweaks you'll ever find. Python gives us a handy way of caching function calls with a simple decorator, functools.lru_cache. Each time you call a function that is decorated with lru_cache, the interpreter checks whether that call was made recently, on a cache that is a dictionary of parameters-result pairs. dict checks are as fast as those of set, and if we have repetitive calls, it's worth looking at this cache before running the code again. from functools import lru_cache @lru_cache(maxsize=16) def my_repetitive_function(value): # pretend this is an extensive calculation return value * 2 for value in range(100): my_repetitive_function(value % 8) The decorator gives the method cache_info, where we can find the statistics about the cache. We can see the eight misses (for the eight times the function was really called), and 92 hits. As we only have eight different inputs (because of the % 8 thing), the cache size was never fully filled. my_repetitive_function.cache_info() CacheInfo(hits=92, misses=8, maxsize=16, currsize=8) Read In addition to these six tips, read a lot, every day. Read books and other people's code. Make code and talk about code. Time, practice, and exchanging experiences will make you a great programmer, and you'll naturally write better Python code. About the author Gabriel Marcondes is a computer engineer working on Django and Python in São Paulo, Brazil. When he is not coding, you can find him at @ggzes, talking about rock n' roll, football, and his attempts to send manned missions to fictional moons.

Now that ES6 has been officially accepted, it’s time to look forward to the next iteration of JavaScript, which is ECMAScript 7. There are many new and exciting features in ES7. Support for asynchronous programming Of all the new features in ES7, the most exciting one, in my view, is the addition of async and await for asynchronous programming, which occurs quite often, especially when you're trying to build applications using Node.js. To explain async and await, it's better you first see an example. Let’s say you have three asynchronous operations, each one dependent on the result returned by the previous one. There are multiple ways you could do that. The most common way to do this is to utilize callbacks. Let’s take a look at the code: myFirstOperation(function(err, firstResult){ mySecondOperation(firstResult, function(err, secondResult){ myThirdOperation(secondResult, function(err, thirdResult){ /* Do something with the third result */ }); }); }); The obvious flaw with this approach is that it leads to a situation known as callback hell. The introduction of promises simplified async programming greatly, so let’s see how the code would look using promises (which were introduced with ES6): myFirstPromise() .then(firstResult => mySecondPromise(firstResult)) .then(secondResult => myThirdPromis(secondResult)) .then(thirdResult =>{ /* Do something with thrid result */ }, err => { /* Handle error */ }); Now, let’s see how to handle these operations using async and await: async function myOperations(){ const firstResult = await myFirstOperation(); const secondResult = await mySecondOperation(firstResult); const thirdResult = await myThirdOperation(secondResult); /* Do something with third result */ }; try { myOperations(); } catch (err) { /* Handle error */ } This looks just like synchronous code? What? Exactly! The use of async and await makes life much simpler, by making async functions seem as if they are synchronous code. Under the hood, though, all of these functions execute in a nonblocking fashion, so you have the benefit of nonblocking async functions, with the simplicity and readability of synchronous code. Brilliant! Object rest and Object spread In ES6, we saw the introduction of array rest and spread operations. These new additions make it easier for you to combine and decompose arrays. ES7 takes this one level further by providing similar functionality for objects. Object rest This is a extension to the existing ES6 destructuring operation. On assignment of the properties during destructuring, if there is an additional ...rest parameter, all the remaining keys and values are assigned to it as another object. For example: const myObject = { lorem : 'ipsum', dolor : 'sit', amet : 'foo', bar : 'baz' }; const { lorem, dolor, ...others } = myObject; // lorem === 'ipsum' // dolor === 'sit' // others === { amet : 'foo', bar : 'baz' } Object spread This is similar to object rest, but is used for constructing objects instead of destructuring them: const obj1 = { amet : 'foo', bar : 'baz' }; const myObject = { lorem : 'ipsum', dolor : 'sit', ...obj1 }; /* myObject === { lorem : 'ipsum', dolor : 'sit', amet : 'foo', bar : 'baz' }; */ This is an alternative way of expressing the Object.assign function already present in ES6. In the precding code, myObject, is a new object, constructed using some properties of obj1 (there is no reference to obj). The equivalent way of doing this in ES6 would be: const myObject = Object.assign({ lorem : 'ipsum', dolor : 'sit' }, obj1); Of course, the object spread notation is much more readable, and the recommended way of assigning new objects, if you choose to adopt it. Observables The Object.observe function is a great new addition for asynchronously monitoring changes made to objects. Using this feature, you will be able to handle any sort of change made to objects, along with seeing how and when that change was made. Let's look at an example of how Object.observe will work: const myObject = {}; Object.observe(myObject, (changes) => { const [{ name, object, type, oldValue }] = changes; console.log(`You tried to ${type} the ${name} property`); }); myObject.foo = 'bar'; //You tried to add the foo property Caveat Although this is a good feature, as of this writing, Object.observe is being tagged as obsolete, which means that this feature could be removed at any time in the future. While it’s still ok to play around and experiment with this, it is recommended not to use it in production systems and larger applications. Additional utility methods There have been additional methods added to the String and Array prototypes: Array.prototype.includes: This checks whether an array includes an element or not: [1,2,3].includes(1); //true String.prototype.padLeft and String.prototype.padRight: 'abc'.padLeft(10); //"abc " 'abc'.padRight(10); //" abc" String.prototype.trimLeft and String.prototype.trimRight: 'n t abc n t'.trimLeft(); //"abc n t" 'n t abc n t'.trimRight(); //"n t abc" Working with ES7 today Many of the features mentioned here are still in the proposal phase, but you can still get started using them in your JavaScript application today! The most common tool used to get started is babel. In case you want to make a browser application, babel is perfect for compiling all of your code to regular ES5. Alternatively, you can use the many babel plugins already available to use babel with your favorite toolbelt or build system. In case you have trouble setting up your project, there are many yeoman generators to help you get started. If you are planning to use ES7 to build a node module or an application in node, there is a yeoman generator available for that as well. About the author Soham Kamani is a Full stack web developer and electronics hobbyist. He is especially interested in JavaScript, Python, and IOT. He can be found on Twitter at @sohamkamani and at sohamkamani.com.

I have a confession to make—I’m an Archer. I’ve taken my fair share of grief for this, with everyone from the network admin that got by on Kubuntu, to the C dev running Gentoo getting in on the fun. As I told the latter, Gentoo is for lazy people; lazy people that want to have their cake and eat it too with respect to the bleeding edge. Given all that, you’ll understand that when I was asked to write this post, my inner Archer balked at the thought of digging through source and documentation to distill this information into something palatable for your average back end developer. I crossed my fingers and wandered over to their GitHub, hoping for a clear roadmap, or at least a bunch of issues milestoned to 6.0.0. I was disappointed even though a lively discussion on dropping XP/Vista support briefly captured my attention. I’ve also been primarily doing front end work for the last few months. While I pride myself on being an IT guy who does web stuff every now and then (someone who cares more about the CI infrastructure than the product that ships), circumstances have dictated that I be in the ‘mockup’ phase for a couple volunteer gigs as well as my own projects, all at once. Convenient? Sure. Not exactly conducive to my generalist, RenaissanceMan self-image, though (as an aside, the annotations in Rap Genius read like a Joseph Ducreux meme generator). Back end framework and functionality has been relegated to the back of my mind. As such, I watched Node bring io.js back into the fold and go to Semver this fall with all the interest of a house cat. A cat that cares more about the mouse in front of him than the one behind the wall. By the time Node 6 drops in April, I’m going to be feasting on back end work. It would behoove me to understand the new tools I’ll be using. So I ignored the Archer on my left shoulder and, and listening to my editor on the right, dug around the source for a bit. Of note is that there’s nothing near as ground breaking as the 4.0.0 release, where Node adopted the ES6 support introduced in io.js 1.0.0. That’s not so much a slight at Node development as a reflection of the timeline of JavaScript development. That said, here’s some highlights from my foray into their issue tracker: Features Core support for Promises I’ve been playing around with Ember for so long that I was frankly surprised that Node didn’t include Core support for Promises. A quick look at the history of Node shows that they originally existed (based on EventEmitters), but that support was removed in 0.2 and ever since server-side promises have been the domain of various npm modules. The PR is still open, so it remains to be seen if the initial implementation of this makes it into 6.0.0. Changing FIPS crypto support In case you’re not familiar with the Federal Information Processing Standard, MDN has a great article explaining it. Node’s implementation left some functionality to be desired, namely that Node compiled with FIPS support couldn’t use nonFIPS hashing algorithms (md5 for one, breaking npm). Node 6 compiled with FIPS support will likely both fix that and flip it’s functionality, such that, by default, FIPS support will be disabled (in Node compiled with it), requiring invocation in order to be used. Replacing C http-parser and DNS resolvers with JS implementations ThesePRs are still open (and have been for almost a year) so I’d say seeing them come in 6.0.0 is unlikely, though the latest discussion on the latter seems centered around CI so I might be wrong on it not making 6.0.0. That being said, while not fundamentally changing functionality too much, it will be cool to see more of the codebase being written in JavaScript. Deprecations fs reevaluation This primarily affects pre-v4 graceful-fs, on which many modules, including major ones such as npm itself and less, are dependent. The problem lies not insofar as these modules are dependent on graceful-fs, as they don’t require v4 or newer. This could lead to breakage for a lot of Node apps whose developers aren’t on their game when it comes to keeping the dependencies up to date. As a set and forget kind of guy, I’m thankful this is just getting deprecated and not outright removed. Removals sys The sys module was deprecated in Node 4 and has always been deprecated in io.js. There was some discussion on its removal last summer, and it was milestoned to 6.0.0. It’s still deprecated as of Node 5.7.0, so we’ll likely have to wait until release to see if it’s actually removed. Nothing groundbreaking, but there is some maturation of the platform in both codebase and deprecations as well as outright removals. If you live on the bleeding edge like I do, you’ll hop on over to Node 6 as soon as it drops in April, but there’s little incentive to make the leap for those taking a more conservative approach to Node development. About the Author Darwin Corn is a Systems Analyst for the Consumer Direct Care Network. He is a mid-level professional with diverse experience in the Information Technology world.

Security often gets forgotten as the technological world quenches its thirst for innovation. Many of the most exciting developments – for both consumers and businesses – have developed way ahead of security concerns. Trends such as IoT and wearables, for example, both present challenges for cybersecurity strategists and white hat hackers. As we bridge the gap between hardware and software new challenges emerge. Arguably, as software becomes more and more embedded in everyday life – both literally and metaphorically – the lines between cybersecurity and traditional security and surveillance become much more blurred too. Recent high profile cases have put cybersecurity on the agenda, even if it remains a subject that’s little understood. The celebrity iCloud hack was a reminder that even the largest systems, built by tech giants and used by hundreds of millions of people, can be attacked. In the UK in 2015 broadband company Talk Talk underwent a massive security attack with 157,000 customers’ data put at risk – the fact that 2 teenage boys were later arrested for the incident only serves to underline that cybersecurity is a strange place. On the one hand demonstrating the disturbing power of a couple of kids with exceptional hacking skills; on the other, the vulnerability of entire systems and infrastructures. Managing security in a world built on Open Source software and cloud The paradoxes of cybersecurity mirror those of the wider software world. On the one hand we’ve never felt more powerful - when it comes to technology, change feels inevitable and exhilarating. Yet the more powerful we feel – as both consumers and programmers, the more we remember the tools that build awesome products, that help us manage huge projects and infrastructures, are largely built and maintained by communities. It is precisely because those tools of apparent progress seem democratic and open that they can be undone, used against the very things they build. While Open Source may be one reason that cybersecurity has become more challenging and complex, it’s worth also thinking about how we use software. The role of cloud in managing software and delivering services in particular has had an impact. Our devices – our digital lives – are no longer intermittently connected to the ‘world wide web’ (a phrase that sounds somewhat dated today) but rather continuously in communication with ‘the cloud’ – information and services are always available. Frictionless user experiences are great for users, but they’re also pretty useful for cybercriminals too. We feel powerless when it comes to security. Yet this lack of power is paradoxically wedded to our contemporary experience of complete control and connectivity and the ability to craft and cultivate our lives online exactly as we want. We want software that is built around our lifestyles with minimum friction, yet that often comes at a price. Consider, for example, what security looked like 10 years ago. The most essential step to being ‘safe’ online was to make sure your firewall was active and your antivirus was up to date. Today that can be difficult, as multiple devices access a range of networks even in the space of a single day (from airport Wi-Fi to mobile data). It’s hard to keep up. The issue isn’t just one for everyday consumers; it’s also a problem for cybersecurity teams developing the products we need. Security is all about stability. This is antithetical to today’s technological ethos. But what can we do to keep systems and infrastructure safe? To keep our information and data secure? How we learned to stop worrying and love hackers But perhaps we’ve found a solution. The emerging phenomenon of the cybersecurity hackathon, in which security experts and friendly hackers are invited to test and expose vulnerabilities, find ways in and around a huge range of software infrastructures. Perhaps the best example of this recently happening was the ‘Hack the Pentagon’ program, the U.S. Government’s ‘bug bounty’, in which more than a thousand security experts (mercenary hackers if you want to be impolite) uncovered hundreds of vulnerabilities in the Pentagon’s software infrastructure. You can find similar events all around the world – organizations whose infrastructure is built on Open Source software are effectively open sourcing their own security capabilities. These sort of events prove that developing security skills (pentesting in particular) can be invaluable, and also a great way to learn more about how software works, and how people have decided to build things. It makes sense. Gone are the days when you could guarantee security with your software package, when you could rely on your contact at Oracle for support. Insofar as most attacks and security risks are always external to a system or an organization, it makes sense to replicate those external threats when trying to identify vulnerabilities and protect yourself. It’s also important to acknowledge that you can’t cover everything internally. It’s easy to be hubristic, but hubris is very often the first sign of weakness. The way forward – UX, IA and security But are hackathons enough? Should cybersecurity in fact be something that we bear greater consideration, as individuals and organizations? Instead of viewing security as a problem to consider at the end of a development process, an irritating inconvenience, by focusing on questions of accessibility and security as design and user experience issues, we can begin to use software in a much smarter and hopefully safer way. For individuals that might mean thinking more carefully about your digital footprint, the data we allow organizations to access. (And yes, maybe we could manage our passwords a little better, but I really didn’t want to include such trite advice…) For businesses and other institutions it may mean aligning cybersecurity with UX and Information Architecture questions. While cybersecurity is definitely a software problem – one which anyone with an inclination towards code should learn more about – it’s also much more than that. It’s a design problem, an issue about where complex software systems interact with the real world in all its complexity and chaos.

If you're a game developer, there's a big chance you're using Unity as your game engine of choice. Unity, and game development itself, has grown quite complex since its first inception, so it's getting hard to keep track of what features are included in a game engine and which ones are not. So let me try to help you with that and see what the latest version of Unity, version 5.3, has in store for you. Multi Scene Editing Let's first talk about what, in my opinion, seems to be the biggest addition to the arsenal of tools available for Unity users: Multi Scene Editing. So what does Multi Scene Editing do? Well, to put it in a simple way, this new editing capability allows a scene to have other scenes in it. With this new feature, you can have a "House" scene that has "Bedroom" and "Kitchen" scenes in it instead of one big "House" scene that has all of the needed objects. You can edit all of them together or just edit one scene separately if you want to focus on editing a particular scene. This new feature doesn't just help you manage a bigger scene, it also helps you working together with another developer, since now each person can edit the same scene separately with less chance of conflict. The following screenshot from the Unity website will give you an idea of what this looks like:   Using Multi Scene Editing is quite simple. All you have to do is drag a scene (or more) to the hierarchy window of your current scene. Unity will then automatically add that scene and all its objects to that open scene. The hierarchy window will show which scene has which objects, so you don't need to worry about being confused over that. Another thing to note is that Unity will set your base scene as the active scene, meaning that any change you made to the scene during runtime will be applied to that scene by default. If you want to access a scene other than the active one, you can do so by using the SceneManager.GetSceneAt() function. Also, Multi Scene Editing supports both 3D and 2D scenes, so you don't have to be concerned about how many dimensions your game has. Speaking of 2D... More Physics2D Tools Unity 5.3 added a bunch of new 2D physics tools that you can use to create various compelling mechanisms for your game. The biggest of the bunch is the BuoyancyEffector2D, a new physics tool that can easily help you make objects floats (or sink) in water based on their weights. While previously you can simulate the floating effect using real world physics equations, this new tool is a tremendous help if you don't want to dabble in physics too much or just want to quickly prototype your game.   Another really useful 2D physics tool that was introduced with this version is the FrictionJoint2D tool. While the 2D physics engine is mostly relevant for platformer type games, sometimes you may also want a little physics for your top down 2D games. So if you, for example, want enemies in your top down brawler game to be able to get knocked back, congratulations, FrictionJoint2D enables you to do exactly that. Other than that, there's also the new FixedJoint2D and RelativeJoint2D tools that allow objects to be glued to each other and the TargetJoint2D tool that allows an object to keep following another object. In-App Purchase Support For quite a long time, Unity has left the implementation of in-app purchase up to third parties, and so plenty of plugins that help developers integrate IAP flourished in the asset store. Well, in the latest version, apparently Unity decided to get more hands-on, adding in-app purchase integration directly to the engine. Right now the integration supports the official app stores for iOS, Android, Mac, and Windows. Unity's in-app purchase integration has a similar approach and functionality to most IAP plugins in the market today, so if you're currently using a third-party plugin for your IAP needs, you're not in any hurry to replace it. That said, Unity is usually pretty quick to adapt to changes in the platforms they support, so you may want to consider switching to Unity's IAP integration if you want to use the latest feature of the platform of your choice. More iOS Integration On their last few product announcements, Apple introduced various additions to the iOS platform, both hardware and software wise. Although previously you needed to use external plugins to utilize those additions, Unity version 5.3 finally supports these new features so you don't have to rely on third-party plugins. These new iOS-specific additions are 3D Touch support, iPad Pro pen support, as well as support for app slicing and on-demand resource features that are heavily used for Apple TV apps. JSON API Another really welcome addition that came with Unity 5.3 is an official API for handling the JSON format. While at a glance, it may not seem that significant, JSON is a format that is widely-used for storing data, so an official support from Unity is really good news. That said, the implementation of JSON API from Unity differs quite a lot from some of the popular JSON libraries out there, so switching the library you used isn't going to be a simple matter. As you can see, there are definitely a lot of new features in Unity 5.3. There are more features that are worth talking about, including the new version of MonoDevelop, the new features on the particle system, or the new samples for the VR project. That said, all of the items I listed here are the features that, in my opinion, you need to know the most. Feel free to check out the changelog yourself if you want to know more about Unity 5.3. About the author Raka Mahesa is a game developer at Chocoarts who is interested in digital technology in general. Outside of work hours, he likes to work on his own projects, with Corridoom VR being his latest released game. Raka also regularly tweets as @legacy99"

We’re living in a world that’s more connected than we once ever thought possible. Even 10 years ago, the idea of our household appliances being connected to our Nokias was impossible to comprehend. But things have changed now and almost every week we seem to be seeing another day-to-day item now connected to the internet. Twitter accounts like @internetofShit are dedicated to pointing out every random item that is now connected to the internet; from smart wallets to video linked toothbrushes to DRM infused wine bottles, but the very real side to all the laughing and caution - For every connected device you connect to your network you’re giving attackers another potential hole to crawl through. This weekend, save 50% on some of our very best IoT titles - or, if ones not enough pick up any 5 features products for $50! Start exploring here. IoT security has simply not been given much attention by companies. Last year two security researchers managed to wirelessly hack into a Jeep Cherokee, first by taking control of the entertainment system and windshield wipers before moving on to disable the accelerator; just months earlier a security expert managed to take over and force a plane to fly sideways by making a single engine go into climb mode. In 2013 over 40 million credit card numbers were taken from US retailer Target after hackers managed to get into the network via the AC company that worked with the retailer. The reaction to these events was huge, along with the multitude of editorials wondering how this could happen… when security experts were wondering in turn how it took so long. The problem until recently was that the IoT was seen mostly as a curio – a phone apps that turns your light on or sets the kettle at the right time was seen as a quaint little toy to mess around with for a bit, it was hard for most to fully realize how it could tear a massive hole in your network security. Plus the speed of which these new gadgets are entering the market is becoming much faster, what used to take 3-4 years to reach the market is now taking a year or less to capitalize on the latest hype; Kickstarter projects by those new to business are being sent out into the world, homebrew is on the rise. To give an example of how this landscape could affect us the French technology institute Eurecom downloaded some 32,000 firmware images from potential IoT device manufacturers and discovered 38 vulnerabilities across 123 products. These products were found in at least 140K devices accessible over the internet. Now imagine what the total number of vulnerabilities across all IoT products on all networks is, the potential number is scarily huge. The wind is changing slowly. In October, the IoT Security Summit is taking place in Boston, with speakers from both the FBI and US Homeland Security playing prominent roles as Speakers. Experts are finally speaking up about the need to properly secure our interconnected devices. As the IoT becomes mainstream and interconnected devices become more affordable to the general public we need to do all we can to ensure that potential security cracks are filled as soon as possible; every new connection is a potential entrance for attackers to break in and many people simply have little to no knowledge of how to improve their computer security. While this will improve as time goes on companies and developers need to be proactive in their advancement of IoT security. Choosing not to do so will mean that the IoT will become less of a tech revolution and more of a failure left on the wayside.

Research suggests we're going to see almost 700 million smartwatches and wearable units shipped to consumers over the next few years. Wearables represent an exciting new frontier for developers - and a potential new cyber security risk. Smartwatches record a surprisingly large amount of data, and that data often isn't very secure. What data do smartwatches collect? Smartwatches are stuffed full of sensors, to monitor your body and the world around you. A typical smartwatch might include any of the following: Gyroscope Accelerometer Light detection Heart rate monitor GPS Pedometer Through SDKs like Apple's ResearchKit, or through firmware like on the FitBit apps can be created to allow a wearable to monitor and collect this very physical personal data. These data collection is benign and useful - but encompasses some very personal parts of an individuals life such as health, daily activities, and even sleeping patterns. So is it secure? Where is the data stored and how can hackers access it? Smart wearables almost always link up to another 'host' device and that device is almost always a mobile phone. Data from wearables is stored and analysed on that host device, and is in turn vulnerable to the myriad of attacks that can be undertaken against mobile devices. Potential attacks include: Direct USB Connection: Physically linking your wearable with a USB port, either after theft or with a fake charging station. Think it's unlikely? So called 'juice jacking' is more common than you might think. WiFi, Bluetooth and Near Field Communication: Wearables are made possible by wireless networks, whether Bluetooth, WiFi, or NFC. This makes them especially vulnerable to the myriad of wireless attacks it is possible to execute - even something a simple as rooting a device over WiFi with SSH. Malware and Web-based Attacks: Mobile devices remain highly vulnerable to attacks from malware and web-based attacks such as StageFright. Why is this data a security risk? You might be thinking "What do I care if some hacker knows how much I walk during the day?" But access to this data has some pretty scary implications. Our medical records are sealed tight for a reason - do you really want a hacker to be able to intuit the state of your health from your heart rate and exercise? What about if they then sell that data to your medical insurer? Social engineering is one of the most used tools of anyone seeking access to a secure system or area. Knowing how a person slept, where they work out, when their heart rate has been elevated - even what sort of mood they might be in - all makes it that much easier for a hacker to manipulate human weakness. Even if we're not a potential gateway into a highly secured organization, this data can be hypothetically used by dodgy advertisers and products to target us when we're at our most vulnerable. For example, 'Freemium' games often have highly sophisticated models for when to push their paid content and turn us into 'whales' who are always buying their product. Access to elements of our biometrics would only make this that much easier. What does this mean? As our lives integrate more and more with information technology, our data moves further and further outside of our own control. Wearables mean the recording of some of our most intimate details - and putting that data at risk in turn. Even when we work to keep it secure, it only takes one momentary lapse to put it at risk to anyone who's ever been interested in seeing it. Information security is only going to get more vital to all of us. Acknowledgements This blog is based on the presentation delivered by Sam Phelps at the Security BSides London 2016.

I love Python, and the Data Analysis library pandas in particular. It's a fantastic library for working with tabular data, contains connectors for many common data formats out of the box, has an excellent performance profile (as it sits on top of NumPy), and has many many common data operations built in. As a data analyst, I use pandas every day, for most of the day. Sometimes I think of ways I could make more use out of it, for example this gist for getting results from Neo4J into a DataFrame. Sometimes, like today, I think about how I could misuse it for a purely procrastinatory purpose. Want to learn Python? This week our Python Fundamentals course is free inside Mapt. It's an accessible introduction that's comprehensive enough to give you the confidence you need to explore Python further. Click here, log in, and go straight to it!  FizzBuzz FizzBuzz is a popular, simple test of a programmer's knowledge and ability, often employed in job interviews to test a candidate's programming/problem solving ability. The basic problem is usually stated as follows: "Write a program that prints the numbers from 1 to 100. But for multiples of three print 'Fizz' instead of the number and for the multiples of five print 'Buzz'. For numbers which are multiples of both three and five print 'FizzBuzz'." There is so much discussion of FizzBuzz on the internet, that it almost seems pointless to discuss it anymore. Regardless of your opinion of it as a tool to test programmer knowledge/ability, it is certainly an interesting, if somewhat basic challenge, and is still approached (with varying degrees of irreverence) to this day. The Challenge Partially inspired by Joel Grus' tongue in cheek article above, and partially out of pure interest, I recently started to ponder how one might go about implementing a FizzBuzz solution using pandas. The intention would be to rely on as few non-pandas operations as possible, whilst still producing legible code. First of all, here's a fairly standard vanilla Python solution to the problem: def fizzbuzz(x): '''returns the fizzbuzz output for an integer x''' output = '' if x % 3 == 0: output += 'Fizz' if x % 5 ==0: output += 'Buzz' if (x % 3) > 0 and (x % 5) > 0: output += str(x) return output for i in range(1,101): print fizzbuzz(i) Now, the most simple way to apply this with pandas would be to just use the apply function on a series of integers: import pandas as pd pd.Series(range(1,100)).apply(fizzbuzz) which is simple, terse and readable, but the logic is still being done outside of pandas. What I'm really after is a way to express that fizzbuzz logic entirely with pandas operations. My first crack at this is displayed below. #create a DataFrame containing all the values we need, all the integers, #and a series of Fizzes and Buzzes we can hack together to make our output values = pd.DataFrame( { 'n':range(1,101), 'fizz':'Fizz', 'buzz':'Buzz' } ) #get columns for each of the seperate output types fizzes = values[values['n'] % 3 == 0]['fizz'] buzzes = values[values['n'] % 5 == 0]['buzz'] ints = values[(values['n'] % 3 > 0) & (values['n'] % 5 > 0)].n.apply(str) #put the columns together as one dataframe again outputs = pd.concat([fizzes,buzzes,ints], axis=1) #for each row, concatenate the non-null values together outputs.apply(lambda x: x[~pd.isnull(x)].str.cat(),axis=1) First, we're taking advantage of pandas' quite clever constructor functions to create a DataFrame with all the values we need. Secondly, we use pandas' expressive filtering syntax to create three separate columns containing ONLY the values we need. The third part is to concatenate these columns together to give us one more dataframe containing only the values we need. This takes advantage of pandas' powerful and extensive indexing capabilities, which returns us a dataframe with a nice contiguous index, and all our values in order. Finally, we use apply again to turn each row into a single string. When you supply axis = 1 to the apply method, it feeds each row to your operation in the form of a Series, which makes it easier to work with the row in question. I was fairly happy with this as a first pass. All the logic for deciding what to print is done with pandas operations, and the flow is fairly clear. It's still pretty long though. Yes it could be condensed down to two (very long and ugly) lines of code, but it still doesn't feel like an 'optimal' solution to this (very silly) problem. We can condense this logic down further, and reach one-liner-nirvana by making even more impractical use of pandas' DataFrame constructor: pd.DataFrame( { 'f':pd.Series('Fizz',index=filter(lambda x: x% 3 == 0, range(1,100))), 'b':pd.Series('Buzz',index=filter(lambda x: x% 5 == 0, range(1,100))), 'n':pd.Series( filter(lambda x: x%3>0 and x%5>0,range(1,100)), index=filter(lambda x: x%3>0 and x%5>0,range(1,100)) ).apply(str) } ).apply( lambda x: x[~pd.isnull(x)].str.cat(), axis=1 ) This really feels like progress. The flow is essentially the same as before, but now we construct the Fizz, Buzz and output Series within the DataFrame constructor. This makes the code more succinct, and also serves the crucial purpose of saving some precious bytes of memory, reducing the evident strain on my 16GB, i5 work PC. We can still do better however. You'll note that the above solution contains a filter() function for the fizzbuzz logic, which is a step backwards (within this already backwards problem), Also, the FizzBuzz lines actually say BuzzFizz, which is very annoying and not actually fixable. Version 0.18 of pandas introduced some really neat changes including an expanded set of arithmetic operations for Timedeltas. Another neat little addition was the addition of the ability to filter Series objects using a callable condition. You can also supply an 'other' argument to return when the conditions aren't met. This allows us to bring that filter logic back into pandas, and create our best/worst solution yet: pd.concat( [ pd.Series('Fizz', index=range(1, 100)).where(lambda x: x.index % 3 ==0, other=''), pd.Series('Buzz', index=range(1, 100)).where(lambda x: x.index % 5 ==0, other=''), pd.Series(range(1,100),index=range(1,100)).where(lambda x:(x % 5 > 0) & (x % 3 > 0), '').apply(str), ], axis=1 ).apply(lambda x: x.str.cat(), axis=1) Ok, Ok, I admit it, I've gone too far. I looked at this daft code, and realised my quest for pandas based purity had driven me to creating this abomination. But hey, I learnt a bit about my favorite library in the process, so what's the harm? If I relax my condition for pandas purity slightly, this can be coaxed into something almost readable: pd.concat( [ pd.Series('Fizz', range(0, 100, 3)), pd.Series('Buzz', range(0, 100, 5)), pd.Series(range(100)).where(lambda x:(x % 5 > 0) & (x % 3 > 0), '').apply(str), ], axis=1 ).apply(lambda x: x.str.cat(), axis=1)[1:] And that's where I decided to stop. It's not going to get any better/worse than that, and besides, I have actual work to do. It's been an entertaining bit of work, and I'm satisfied this solution is about as small as I'm going to get within the ridiculous constraints I set myself. Also, I can hear the ardent code golfers screaming at me that NONE of my pandas solutions are valid, because they all print an index column as well as the fizzbuzz values. I don't think there is a way of overcoming this purely in pandas, so you'll have to settle for just wrapping any of them with 'n'.join(...) Performance So, we have several methods for fizzbuzzing with pandas. The next obvious question is about scalability and performance. What if you're interviewing to be a Big Data Engineer at Google, and they ask you for the fastest, most scalable, general fizzbuzz solution in your arsenal? Let's take our functions above for a test drive. We'll encapsulate each as a function, then see how they scale as we ask for ever larger ranges. I'm using a quick and dirty method of timing this as I don't actually care that much. def fizzbuzz(x): '''returns the fizzbuzz output for an integer x''' output = '' if x % 3 == 0: output += 'Fizz' if x % 5 ==0: output += 'Buzz' if (x % 3) > 0 and (x % 5) > 0: output += str(x) return output #our vanilla solution def fb_vanilla(rng): return map(fizzbuzz, range(1, rng+1)) #our trivial pandas solution def fb_pandas_vanilla(rng): return pd.Series(range(1, rng+1)).apply(fizzbuzz) #I'm going to skip the first big pandas solution, this is pretty much identical #our second pandas solution, down to one line def fb_pandas_long(rng): return pd.DataFrame( { 'f':pd.Series('Fizz',index=filter(lambda x: x% 3 == 0, range(1,rng+1))), 'b':pd.Series('Buzz',index=filter(lambda x: x% 5 == 0, range(1,rng+1))), 'n':pd.Series( filter(lambda x: x%3>0 and x%5>0,range(1,rng+1)), index=filter(lambda x: x%3>0 and x%5>0,range(1,rng+1)) ).apply(str) } ).apply( lambda x: x[~pd.isnull(x)].str.cat(), axis=1 ) #our more succinct, pandas only solution. def fb_pandas_shorter(rng): return pd.concat( [ pd.Series('Fizz', index=range(1, rng+1)).where(lambda x: x.index % 3 ==0, other=''), pd.Series('Buzz', index=range(1, 100)).where(lambda x: x.index % 5 ==0, other=''), pd.Series(range(1,rng+1),index=range(1,rng+1)).where(lambda x:(x % 5 > 0) & (x % 3 > 0), '').apply(str), ], axis=1 ).apply(lambda x: x.str.cat(), axis=1) #our shortest solution, relying on some non-pandas stuff def fb_pandas_shortest(rng): return pd.concat( [ pd.Series('Fizz', range(0, rng+1, 3)), pd.Series('Buzz', range(0, rng+1, 5)), pd.Series(range(rng+1)).where(lambda x:(x % 5 > 0) & (x % 3 > 0), '').apply(str), ], axis=1 ).apply(lambda x: x.str.cat(), axis=1)[1:] #Let's do some testing! functions = [ fb_vanilla, fb_pandas_vanilla, fb_pandas_long, fb_pandas_shorter, fb_pandas_shortest ] times = {x.__name__:[] for x in functions} tests = range(1,1000,100) from time import time for i in tests: for x in functions: t1 = time() _ = x(i) t2 = time() times[x.__name__].append(t2-t1) results = pd.DataFrame(times, index = tests) Well, so far so terrible. The first, longest solution actually scales O(n), which I find hilarious for some reason. The rest of my attempts fare a little better, but nothing compares to the vanilla python solution, which doesn't even show up on this chart, because it completes in <1ms. Let's discard that long solution, and try an even more strenuous test, up to 100,000. That seems pretty conclusive. Get me Google on the phone, I have solved this problem. When should this be used? NEVER. Like, seriously, those results are beyond terrible. I was expecting the pandas solutions to fall down compared to the vanilla python solutions, but this is ridiculous. If you're asked to implement fizzbuzz in your next interview, the only possible reason you would have for using any of these is because you're a flippant contrarian like me. Having said that, it's important to note that I still love pandas, and don't blame these results on the fantastic devs there. This is thoroughly not what the library is designed for, and I'm only doing this to test my own fluency with the tool. Overall I think I acheived my goal with this. I made myself chuckle a few times, and I learnt a few things about pandas in the process. If you have any suggestions for optimisations to this code, reach out in the comments or on Twitter. Equally, if you have any thoughts on why specifically these pandas functions are SOOO much slower than the vanilla solution, I'd be interested to hear them.

This is the final installation of a five-part series that began with a guide to container creation [link to part 1] and cloud deployment [link to part 2] and progressed to application layering [link to part 3] and deploying tiered applications to a cluster [link to part 4]. This series will now wrap up by focusing less on the Docker ecosystem to look at other clustered deployment options. This is largely beyond the scope of a series of posts focused on Docker. However, while working on this blog post series, I also worked with CoreOS. The beauty of CoreOS is that it fully integrates with the core Docker Engine (though as of this writing, RKT, their own container runtime, was just declared production ready. However, I'm sure they'll be pushing users in this direction as development continues). The CoreOS ecosystem consists of these primary components: CoreOS: This is the base operating system etcd: This is the distributed key-value store fleet: This is the coordinating systemd across the cluster flannel: This connects it all together There's a lot more than this, but those are the heavy-hitting aspects of the distro. Let's take a look at each of the parts individually in the context of the Taiga application used in Parts 3 and 4. I've once again branched the docker-taiga repo, so run git pull to ensure that you're working with the latest material. Then, run git checkout coreos, and you'll be ready to follow along. Like part four, there's a deploy.sh script in the application root, and this time, if you're running Linux with KVM/libvirt and at least 4 GB of available RAM, then you can kick it off and watch the magic of a highly available Taiga cluster launch on your very own computer. CoreOS CoreOS is a Gentoo-based Linux distro designed for one purpose: container deployments. While it can be run in a solo instance, it really shines in a cluster and integrates with a host of associated projects to make this happen. There's not much of interest in the OS itself, besides that it's deployed in a way that is different from what you're likely used to if you come from the Red Hat / Ubuntu server world. The machine is provisioned with a YAML file called a cloud-config and can be booted up in a number of different ways, from your favorite cloud provider's SDK to a simple install script. etcd etcd is a simple key-value store (think Redis) specifically tailored for use in CoreOS. It's meant to house the data needed across the entirety of the system—database connection information, feature flags, and so on—as well as handle the system coordination itself, including determining the leader machine / container election. It's a versatile tool, and for a full use case outside the one demonstrated in deploy.sh, I highly recommend looking at CoreOS's list of projects using etcd. fleet The fleet tool is probably my favorite of the coordinating distributed systemd bunch. The fleet tool is where CoreOS shines in it's ease of achieving high availability because it lets you use the now-familiar unit files of systemd to schedule processes on one, some, or all of the machines in a cluster. Perhaps its simplest application is in running containers, and that's only scratching the surface in what it can be used to accomplish. One of the cooler use cases is a simplified service discovery, which helps the cluster stay coordinated as well as making it scalable. A form of the sidekick method is demonstrated in some of the unit files that deploy.sh references. flannel Looking at flannel is where I started getting uncomfortable, and then I was immediately assuaged. In a previous job I had to manage firewalls and switches, and at no point did my pretend network engineer alter ego feel comfortable troubleshooting network issues. I used to joke that my job was watching Cisco SmartNet do it for me, but flannel takes all that and simplifies it, running an agent on each node that controls the subnet for containers on this host and records it in etcd to ensure the configuration isn't lost in case this node goes down for whatever reason. Port mapping then becomes the trivial process of letting flannel route it for you given a minimal amount of configuration. Further reading Of course, we covered all this without even touching on Kubernetes, which uses some of the CoreOS toolset itself. In fact, the two pair pretty well together, but you could write a whole book on Kubernetes alone. If you want to read more about Kubernetes, start with this excellent (if brief) guest blog on CoreOS's website about integrating Ansible with CoreOS deployments. After this, feel free to reference this post and the series as a whole as you descend into the rabbit hole that is highly available, containerized, and tiered applications. Good luck! About the Author Darwin Corn is a systems analyst for the Consumer Direct Care Network. He is a mid-level professional with diverse experience in the information technology world.

We say it a lot, but Python really is a versatile language that can be applied to many different purposes. Web developers, data analysts, security pros - there's an impressive range of challenges that can be solved by Python. So, what exactly should you be learning to do with this great language to really get the most out of it?   Writing Python What's the most important thing to learn with Python? How to write it. As Python becomes the popular language of choice for most developers, there is an increasing need to learn and adopt it on different environments for different purposes. The Beginning Python video course focuses on just that. Aimed at a complete novice with no previous programming experience in Python, this course will guide the readers every step of the way. Starting with the absolute basics like understanding of variables, arrays, and strings, the course goes on teach the intricacies of Python. It teaches how you can build your own functions making use of the existing functions in Python. By the end, the course ensures that you have a strong foundation of the programming concepts in Python. Design Patterns As Python matures from being used just as a scripting language and into enterprise development and data science, the need for clean, reusable code becomes ever more vital. The modern Python developer cannot go astray with tried and true design patterns for Python when they want to write efficient, reliable Python code. The second edition of Learning Python Design Patterns is stuffed with rich examples of design pattern implementation. From OOP to more complex concepts, you'll find everything you need to improve your Python within. Machine Learning Design We all know how powerful Python is for machine learning - so why are your results proving sub-par and inaccurate? The issue is probably not your implementation, but rather with your system design. Just knowing the relevant algorithms and tools is not enough for a really effective system - you need the right design. Designing Machine Learning Systems with Python covers various machine learning designing aspects with the help of real-world data sets and examples and will enable you to evaluate and decide the right design for your needs. Python for the Next Generation Python was built to be simple, and it's the perfect language to get kids coding. With programmers getting younger and younger these days, get them learning with a language that will serve them well for life. In Python for Kids, kids will create two interesting game projects that they can play and show off to their friends and teachers, as well as learn Python syntax, and how to do basic logic building. Distributed Computing What do you do when your Python application takes forever to give the output? Very heavy computing results in delayed response or, sometimes, even failure. For special systems that deal with a lot of data and are mission critical, the response time becomes an important factor. In order to write highly available, reliable, and fault tolerant programs, one needs to take aid of distributed computing. Distributed Computing with Python will teach you how to manage your data intensive and resource hungry Python applications with the aid of parallel programming, synchronous and asynchronous programming, and many more effective techniques. Deep Learning Python is at the forefront of the deep learning revolution - the next stage of machine learning, and maybe even a step towards AI. As machine learning becomes a mainstream practice, deep learning has taken a front seat among data scientists. The Deep Learning with Python video course is a great stepping stone in entering the world of deep learning with Python -- learn the basics, clear your concepts, and start implementing efficient deep learning for making better sense of data. Get all that it takes to understand and implement Python deep learning libraries from this insightful tutorial. Predictive Analytics With the power of Python and predictive analytics, you can turn your data into amazing predictions of the future. It's not sorcery, just good data science. Written by Ashish Kumar, a data scientist at Tiger Analytics, Learning Predictive Analytics with Python is a comprehensive, intermediate-level book on Predictive Analytics and Python for aspiring data scientists. Internet of Things Python's rich libraries of data analytics, combined with its popularity for scripting microcontroller units such as the Raspberry Pi and Arduino, make it an exceptional choice for building IoT. Internet of Things with Python offers an exciting view of IoT from many angles, whether you're a newbie or a pro. Leverage your existing Python knowledge to build awesome IoT project and enhance your IoT skills with this book.  

Last November, Google open sourced its shiny Machine Intelligence package, promising a simpler way to develop deep learning algorithms that can be deployed anywhere, from your phone to a big cluster without a hassle. They even take advantage of running over GPUs for better performance. Let's Give It a Shot! First things first, let's install it: # Ubuntu/Linux 64-bit, CPU only (GPU enabled version requires more deps): $ sudo pip install --upgrade https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.7.1-cp27-none-linux_x86_64.whl # Mac OS X, CPU only:$ sudo easy_install --upgrade six$ sudo pip install --upgrade https://storage.googleapis.com/tensorflow/mac/tensorflow-0.7.1-cp27-none-any.whl We are going to play with the old-known iris dataset, where we will train a neural network to take dimensions of the sepals and petals of an iris plant and classify it between three different types of iris plants: Iris setosa, Iris versicolour, and Iris virginica. You can download the training CSV dataset from here. Reading the Training Data Because TensorFlow is prepared for cluster-sized data, it allows you to define an input by feeding it with a queue of filenames to process (think of MapReduce output shards). In our simple case, we are going to just hardcode the path to our only file: import tensorflow as tf def inputs(): filename_queue = tf.train.string_input_producer(["iris.data"]) We then need to set up the Reader, which will work with the file contents. In our case, it's a TextLineReader that will produce a tensor for each line of text in the dataset: reader = tf.TextLineReader()key, value = reader.read(filename_queue) Then we are going to parse each line into the feature tensor of each sample in the dataset, specifying the data types (in our case, they are all floats except the iris class, which is a string). # decode_csv will convert a Tensor from type string (the text line) in # a tuple of tensor columns with the specified defaults, which also # sets the data type for each column sepal_length, sepal_width, petal_length, petal_width, label = tf.decode_csv(value, record_defaults=[[0.0], [0.0], [0.0], [0.0], [""]]) # we could work with each column separately if we want; but here we # simply want to process a single feature vector containing all the # data for each sample. features = tf.pack([sepal_length, sepal_width, petal_length, petal_width]) Finally, in our data file, the samples are actually sorted by iris type. This would lead to bad performance of the model and make it inconvenient for splitting between training and evaluation sets, so we are going to shuffle the data before returning it by using a tensor queue designed for it. All the buffering parameters can be set to 1500 because that is the exact number of samples in the data, so will store it completely in memory. The batch size will also set the number of rows we pack in a single tensor for applying operations in parallel: return tf.train.shuffle_batch([features, label], batch_size=100, capacity=1500, min_after_dequeue=100)   Converting the Data Our label field on the training dataset is a string that holds the three possible values of the Iris class. To make it friendly with the neural network output, we need to convert this data to a three-column vector, one for each class, where the value should be 1 (100% probability) when the sample belongs to that class. This is a typical transformation you may need to do with input data. def string_label_as_probability_tensor(label): is_setosa = tf.equal(label, ["Iris-setosa"]) is_versicolor = tf.equal(label, ["Iris-versicolor"]) is_virginica = tf.equal(label, ["Iris-virginica"]) return tf.to_float(tf.pack([is_setosa, is_versicolor, is_virginica]))   The Inference Model (Where the Magic Happens) We are going to use a single neuron network with a Softmax activation function. The variables (learned parameters of our model) will only be the matrix weights applied to the different features for each sample of input data. # model: inferred_label = softmax(Wx + b) # where x is the features vector of each data example W = tf.Variable(tf.zeros([4, 3])) b = tf.Variable(tf.zeros([3])) def inference(features): # we need x as a single column matrix for the multiplication x = tf.reshape(features, [1, 4]) inferred_label = tf.nn.softmax(tf.matmul(x, W) + b) return inferred_label Notice that we left the model parameters as variables outside of the scope of the function. That is because we want to use those same variables both while training and when evaluating and using the model. Training the Model We train the model using backpropagation, trying to minimize cross entropy, which is the usual way to train a Softmax network. At a high level, this means that for each data sample, we compare the output of the inference with the real value and calculate the error (how far we are). Then we use the error value to adjust the learning parameters in a way that minimizes that error. We also have to set the learning factor; it means for each sample, how much of the computed error we will apply to correct the parameters. There has to be a balance between the learning factor, the number of learning loop cycles, and the number of samples we pack tighter in the same tensor in batch; the bigger the batch, the smaller the factor and the higher the number of cycles. def train(features, tensor_label): inferred_label = inference(features) cross_entropy = -tf.reduce_sum(tensor_label*tf.log(inferred_label)) train_step = tf.train.GradientDescentOptimizer(0.001) .minimize(cross_entropy) return train_step Evaluating the Model We are going to evaluate our model using accuracy, which is the ratio of cases where our network identifies the right iris class over the total evaluation samples. def evaluate(evaluation_features, evaluation_labels): inferred_label = inference(evaluation_features) correct_prediction = tf.equal(tf.argmax(inferred_label, 1), tf.argmax(evaluation_labels, 1)) accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32)) return accuracy Running the Model We are only left to connect our graph and run it in a session, where the defined operations are actually going to use the data. We also split our input data between training and evaluation around 70%:30%, and run a training loop with it 1,000 times. features, label = inputs() tensor_label = string_label_as_probability_tensor(label) train_step = train(features[0:69, 0:4], tensor_label[0:69, 0:3]) evaluate_step = evaluate(features[70:99, 0:4], tensor_label[70:99, 0:3]) with tf.Session() as sess: sess.run(tf.initialize_all_variables()) # Start populating the filename queue. coord = tf.train.Coordinator() threads = tf.train.start_queue_runners(coord=coord) for i in range(1000): sess.run(train_step) print sess.run(evaluate_step) # should print 0 => setosa print sess.run(tf.argmax(inference([[5.0, 3.6, 1.4, 0.2]]), 1)) # should be 1 => versicolor print sess.run(tf.argmax(inference([[5.5, 2.4, 3.8, 1.1]]), 1)) # should be 2 => virginica print sess.run(tf.argmax(inference([[6.9, 3.1, 5.1, 2.3]]), 1)) coord.request_stop() coord.join(threads) sess.closs() If you run this, it should print an accuracy value close to 1. This means our network correctly classifies the samples in almost 100% of the cases, and also we are providing the right answers for the manual samples to the model. Conclusion Our example was very simple, but TensorFlow actually allows you to do much more complicated things with similar ease, such as working with voice recognition and computer vision. It may not look much different than using any other deep learning or math packages, but the key is the ability to run the expressed model in parallel. Google is willing to create a mainstream DSL to express data algorithms focused on machine learning, and they may succeed in doing so. For instance, although Google has not yet open sourced the distributed version of the engine, a tool capable of running Tensorflow-modeled graphs directly over an Apache Spark cluster was just presented at the Spark Summit, which shows that the community is interested in expanding its usage. About the author Ariel Scarpinelli is a senior Java developer in VirtualMind and is a passionate developer with more than 15 years of professional experience. He can be found on Twitter at @ triforcexp.

There's this great new library called redux that has been making rounds recently for all of the right reasons. But, why are developers so crazy about redux, and why should you consider using redux in your next application? This post will explain what makes redux a good choice as we create a small application using redux and its principles. Redux is based on three main principles; they are as  follows: Every app has a single store: The "store" is where the state of your application is stored. Your app must have its entire state in only one store. This is because there has to be a single source of truth that renders the rest of your application. What this means is that if there’s something that doesn't look right in your app, it's easier to track the source of the bug because you know exactly where the state of each component is coming from. The application state is immutable: Once you have set your application state, it cannot be mutated again. This doesn't mean your application can't be dynamic. State can only be changed through actions and reducers (explained next), in which case you have to recompute the new state of the application each time and not change the existing state. The immutable state is better for your application because every time there is a new state, your application gets notified and can re-render accordingly. In this way, you are guaranteed to have your application show a visual representation of your state at any point in time. All reducers are pure functions: As stated before, the only way you can change the state of your application is by recomputing the new state from the old state. This is done with a reducer, which is nothing more than a function that takes two arguments (that is, the previous state and the action required) and returns the new application state. The most important concept here is that all reducers must be pure functions. If your reducer is doing anything outside the function's scope, it's possible that you're doing it wrong. Writing reducers The standard format of a reducer is as follows: const myReducer = (state = defaultValue, action) => { /* perform some calculations based on action and old state. newState !== state */ return newState; }; Using this format, let’s write a reducer for a simple counter: const counter = (state = 0, action) => { const { type } = action; switch (type) { case 'INCREMENT': return state + 1; case 'DECREMENT': return state + 2; default: return state; } }; The state variable is the original state passed to the reducer—we give a default value of 0 just in case (because this is the first time the reducer is called). The action argument is an object that contains a type attribute describing the kind of change we want in our state. If the action is of the INCREMENT type, we return the state increased by one and decrease the state by one for the DECREMENT type. If the type of action passed is unrecognized, we just return the state as it is. This is an important concept to remember because it will become very important once the application grows in size. Writing an application using the reducer So far, there has been no mention of redux, only of reducers. We now need redux as a glue to bridge our business logic (the counter reducer) to the store and the application state. In order to make our application, we will use npm and ES6 modules. You can bootstrap a project easily using a yeoman generator like this one: Install redux and react using the following: npm install --save redux Create the counter store: import { createStore } from 'redux'; const store = createStore(counter); In our html file, we will add a simple interface for our counter: <html> <head> <title>My App</title> </head> <body> <div id="counter-display"> </div> <button id="counter-increment"> + </button> <button id="counter-decrement"> - </button> <script src="bundle.min.js"></script> </body> </html> Next, let’s create a render method and subscribe our store to it such that it is called every time the state of the store is changed: const counterDisplay = document.getElementById('counter-display'); const render = () => { counterDisplay.innerHTML = store.getState(); }; store.subscribe(render); render(); We also call the render method once in the beggining to render the app initially. Now, we will add event listeners to the increment and decrement buttons to dispatch events every time they are clicked: const incrementButton = document.getElementById('counter-increment'); const decrementButton = document.getElementById('counter-decrement'); incrementButton.addEventListener('click', ()=>{ store.dispatch({type : 'INCREMENT'}); }); decrementButton.addEventListener('click', ()=>{ store.dispatch({type : 'DECREMENT'}); }); Now we have a fully functioning counter. The data flow in/out counter is as follows: The user clicks a button (increment or decrement). The event listener dispatches an event, with a type of either INCREMENT or DECREMENT, based on the button clicked. The reducer re-computes the state of the store depending on the action type. Since there is a new state, the render function, which was subscribed to the state, is called. The render method gets the current state from the store and changes the contents of the DOM. The source code for this application can be found here, and the working example can be seen here. Redux developer tools Redux is so great because of the many developer tools that it makes available. This one is written by the creator of redux himself. A few of the reasons you should consider incorporationg developer tools into your development are as follows: They provide a way to constantly monitor your state. No more pesky console.logs to check what your current state is. You can see exactly which action changed the state. There’s no more guesswork. If the state was changed, you now know exactly when and why it was changed. You can change the past. Yes, you read that right! The redux developer tools give you the option of removing an action you may have performed some time ago and re-computing the state to show you the current state of the application, as if that action had never been performed at all. For small scale applications, redux devolopment tools provide an easy and convenient way to debug and inspect your application, and for larger applications, I would go so far as to say that they are required. About the author Soham Kamani is a Full Stack web developer and electronics hobbyist. He is especially interested in JavaScript, Python, and IOT. HisTwitter handle is @sohamkamani, and he can also be found here.

Just a few months ago we were talking about the latest iteration of PacktLib. And it seems plenty of people love it. But we’re not done yet; behind the scenes here at Packt we’ve been working hard to add some really exciting new features, add a little extra style, to give you a more complete and a more enjoyable learning experience. If you haven’t already, give it a try – whoever you are, and whatever your experience of Packt, we’re confident that you’re going to find more than what you bargained for. Once you subscribe you’ll have access to some of our most popular content, including Python Machine Learning, React JS by Example and Switching to Angular 2. With the software landscape rapidly changing, by subscribing you’ll have everything you need to navigate the challenges of tomorrow. And, of course, simply build better software. If you want to try it, the first two chapters of every single title in our collection (that’s more than 3,500 eBooks) are available for you to read for free. That means you can take a look at a huge selection of content, as well as experiencing our neat, and endlessly usable reader for yourself. If you want to subscribe, make sure you do it soon! We’ll soon be making more changes, which means we will be restructuring our price – so why not dive in and explore our library today? Your PacktLib development journey begins here.

If you're a Unity developer looking to get into VR development, then you're in luck because Unity is definitely going all-in on the virtual reality front. With the recent announcements at Vision VR/AR Summit 2016, Unity has promised built-in support for all major virtual reality platforms, which are Oculus Rift (including GearVR), Playstation VR, SteamVR, and Google Cardboard. Unity is also currently developing an experimental version of the Unity editor that can be used inside of VR. Given how there is one game engine that supports most VR platforms, there has never been a better time to get your feet wet in VR development. So let's do exactly that; let's get started with VR using Unity. And don't worry; I'll be here to help you every step of the way. A journey of a thousand miles begins with a single step, so let's do some simple stuff first. There are two basic topics that we're going to cover this time: head tracking and interaction in VR. While those sound pretty basic, they are the fundamental components of a virtual reality experience, so it's best to understand them in the beginning. Requirement There are a couple of requirements that need to be satisfied before we start, however. We're going to need: Unity 5.3 or above Windows 7 or above Oculus Rift device (optional) Oculus Runtime 0.8.0.0 (only if you have an Oculus Rift) While it's recommended to have an Oculus Rift ready when you're developing a VR app, it is not required. Starting out All right then, let's get started. I'm going to assume that you're already familiar with programming and Unity in general, so we won’t go into too much detail with that. Let's make a new Unity 3D project. Once you've made a new project, you need to turn on VR support by going to Edit > Project Settings > Player. On the Other Settings tab, check the Virtual Reality Supported box. Don't forget to make sure that your selected build platform is PC, Mac, and Linux. The next step is to create a scene and add some objects there. Any objects will do, but in my case, I just added a cube at the center of the scene. Then we create an empty GameObject and add (or move) the main camera as its child object. This part is important—the main camera has to be a child of another object because when you're developing for VR, you cannot move the camera object by yourself. The VR system will track your movement and adjust the camera position accordingly, overriding any change you made to the camera position. So, to have your camera moving, you'll have to modify the position of the camera's parent object instead of the camera itself. Now, if you have an Oculus Rift in your hand, you're set and ready to go! All you need to do is simply connect the device to your PC, and when you press play, your head movement will automatically be tracked and translated to the in-game camera and you will be able to see your scene in VR on your Oculus Rift. Don't fret if there's no Oculus Rift available for you to use. We'll just have to simulate head movements using this script (by Peter Koch from talesfromtherift). Copy that script to your project and attach it to your main camera. Now, if you play the project in the editor, you can rotate the camera freely by holding the Alt button and moving your cursor around. Interaction Okay, time to step things up a notch. It's not really an app if you can't interact with it, so let's inject some interactivity into the project. We'll add the most basic interaction in virtual reality—gazing. To add gazing to our project, we're going to need this script. Copy and attach it to the main camera. Now when you're playing the project, if you look at a GameObject that has a Collider, the object will slowly turn red. If you clicked while you're looking at an object, that object will be teleported to a random position. Interaction is fun, isn't it? Well, let's dive deeper into the script that enables that gazing interaction. Basically, at every frame, the script will check whether you're looking at an object by casting a ray forward from the camera and seeing whether it hits an object or not. if (Physics.Raycast(new Ray(transform.position, transform.forward), out hit, GAZE_LENGTH)) When the cast ray hits an object, it will check whether it's the same object as the previous frame or not. If it's the same object, a timer will be updated and the script will change the color of the object according to that timer. //Increase gaze time mGazeTime += Time.deltaTime; if (mGazeTime > MAX_GAZE_DURATION) mGazeTime = MAX_GAZE_DURATION; //Recolor float color = (MAX_GAZE_DURATION - mGazeTime) / MAX_GAZE_DURATION; ColorObject(mObject, new Color(1f, color, color)); Interacting via clicks is just as simple. After the script has detected that there's an object, it checks whether the left mouse button is clicked or not. If the button is clicked, the script will then move the object to another position. if (mObject != null && Input.GetMouseButtonUp(0)) { //Move object elsewhere float newX = Random.Range(-8, 8f); float newY = Random.Range(-2f, 2f); float newZ = Random.Range(0, 3f); mObject.transform.position = new Vector3(newX, newY, newZ); } Right now the script, while functional, is very basic. What if you want to have different objects behave differently when they're being looked at? Or what if you want the script to interact with only several objects? Well, I figure I'll leave all that to you as exercise. This concludes the beginning of our journey into VR development. While we only scratched the surface of the development process in this post, we've learned enough stuff to actually make a functioning VR app or game. If you're interested in working on more VR projects, check out the Unity VR samples. They have a bunch of reusable codes that you can use for your VR projects. Good luck! About this author Raka Mahesa is a game developer at Chocoarts  who is interested in digital technology in general. Outside of work hours, he likes to work on his own projects, with Corridoom VR  being his latest released game. Raka also regularly tweets as @legacy99

It's been nearly four months since TensorFlow, Google's computation graph machine learning library, was open sourced, and the momentum from its launch is still going strong. Over the time, both Microsoft and Baidu have released their own deep-learning libraries (CNTK and warp-ctc, respectively), and the machine learning arms race has escalated even further with Yahoo open sourcing CaffeOnSpark. Google hasn't been idle, however, and with the recent releases of TensorFlow Serving and the long awaited distributed runtime, now is the time for businesses and individual data scientists to ask: is it time to commit to TensorFlow? TensorFlow's most appealing features There are a lot of machine learning libraries available today—what makes TensorFlow stand out in this crowded space? 1. Flexibility without headaches TensorFlow heavily borrows concepts from the more tenured machine learning library Theano. Many models written for research papers were built in Theano, and its composable, node-by-node writing style translates well when implementing a model whose graph was drawn by hand first. TensorFlow's API is extremely similar. Both Theano and TensorFlow feature a Python API for defining the computation graph, which then hooks into high performance C/C++ implementations of mathematical operations. Both are able to automatically differentiate their graphs with respect to their inputs, which facilitates learning on complicated neural network structures and both integrate tightly with Numpy for defining tensors (n-dimensional arrays). However, one of the biggest advantages TensorFlow currently has over Theano (at least when comparing features both Theano and TensorFlow have) is its compile time. As of the time of writing this, Theano's compile times can be quite lengthy and although there are options to speed up compilation for experimentation, they come at the cost of a slower output model. TensorFlow's compilation is much faster, which leads to less headaches when trying out slightly different versions of models. 2. It's backed by Google (and the OSS community) At first, it may sound more like brand recognition than a tangible advantage, but when I say it's 'backed' by Google, what I mean is that Google is seriously pouring tons of resources into making TensorFlow an awesome tool. There is an entire team at Google dedicated on maintaining and improving the software steadily and visibly, while simultaneously running a clinic on how to properly interact with and engage the open source community. Google proved itself willing to adopt quality submissions from the community as well as flexible enough to adapt to public demands (such as moving the master contribution repository from Google's self-hosted Gerrit server to GitHub). These actions combined with genuinely constructive feedback from Google's team on pull-requests and issues helped make the community feel like this was a project worth supporting. The result? A continuous stream of little improvements and ideas from the community while the core Google team works on releasing larger features. Not only does TensorFlow recieve the benefits of a larger contributor base because of this, it also is more likely to withstand user decay as more people have invested time in making TensorFlow their own. 3. Easy visualizations and debugging with TensorBoard TensorBoard was the shiny toy that shipped on release with the first open source version of TensorFlow, but it's much more than eye candy. Not only can you use it as a guide to ensure what you've coded matches your reference model, but you can also keep track of data flowing through your model. This is especially useful when debugging subsections of your graph, as you can go in and see where any hiccups may have occurred. 4. TensorFlow Serving cuts the development-deployment cycle by nearly half The typical life cycle of machine learning models in the business world is generally as follows: Research and develop a model that is more accurate/faster/more descriptive than the previous model Write down the exact specifications of the finalized model Recreate the model in C++/C/Java/some other fast, compiled language Push the new model into deployment, replacing the old model Repeat On release, TensorFlow promised to "connect research and production." However, the community had to wait until just recently for that promise to come to fruition with TensorFlow Serving. This software allows you to run it as a server that can natively run models built in TensorFlow, which makes the new life cycle look like this: Research and develop a new model Hook the new model into TensorFlow Serving Repeat While there is overhead in learning how to use TensorFlow Serving, the process of hooking up new models stays the same, whereas rewriting new models in a different language is time consuming and difficult. 5. Distributed learning out of the box The distributed runtime is one of the newest features to be pushed to the TensorFlow repository, but it has been, by far, the most eagerly anticipated aspect of TensorFlow. Without having to incorporate any other libraries or software packages, TensorFlow is able to run distributed learning tasks on heterogenous hardware with various CPUs and GPUs. This feature is absolutely brand new (it came out in the middle of writing this post!), so do your research on how to use it and how well it runs. Areas to look for improvement TensorFlow can't claim to be the best at everything, and there are several sticking points that should be addressed sooner rather than later. Luckily, Google has been making steady improvements to TensorFlow since it was released, and I would be surprised if most of these were not remedied within the next few months. Runtime speed Although the TensorFlow team promises deployment worthy models from compiled TensorFlow code, at this time, its single machine training speed lags behind most other options. The team has made improvements in speed since its release, but there is still more work to be done. In-place operations, a more efficient node placement algorithm, and better compression techniques could help here. Distributed benchmarks are not available at this time—expect to see them after the next official TensorFlow release. Pre-trained models Libraries such as Caffe, Torch, and Theano have a good selection of pre-trained, state-of-the-art models that are implemented in their library. While Google did release a version of its Inception-v3 model in TensorFlow, it needs more options to provide a starting place for more types of problems. Expanded distributed support Yes, TensorFlow did push code for it's distributed runtime, but it still needs better documentation as well as more examples. I'm incredibly excited that it's available to try out right now, but it's going to take some time for most people to put it into production. Interested in getting up and running with TensorFlow? You'll need a primer on Python. Luckily, our Python Fundamentals course in Mapt gives you an accessible yet comprehensive journey through Python - and this week it's completely free. Click here, login, then get stuck in... The future Most people want to use software that is going to last for more than a few months—what does the future look like for TensorFlow? Here are my predictions about the medium-term future of the library. Enterprise-level distributions Just as Hadoop has commercial distributions of its software, I expect to see more and more companies offering supported suites that tie into TensorFlow. Whether they have more pre-trained models built on top of Keras (which already supports a TensorFlow backend), or make TensorFlow work seamlessly with a distributed file system like Hadoop, I forsee a lot of demand for enterprise features and support with TensorFlow. TensorFlow's speed will catch up (and most users won't need it) As mentioned earlier, TensorFlow still lags behind many other libraries out there. However, with the improvements already made; it's clear that Google is determined to make TensorFlow as efficient as possible. That said, I believe most applications of TensorFlow won't desperately need the speed increase. Of course, it's nice to have your models run faster, but most businesses out there don't have petabytes of useful data to work with, which means that model training usually doesn't take the "weeks" that we often see claimed as training time. TensorFlow is going to get easier, not more difficult, over time While there are definitely going to be many new features in upcoming releases of TensorFlow, I expect to see the learning curve of the software go down as more resources, such as tutorials, examples, and books are made available. The documentation's terminology has already changed in places to be more understandable; navigation within the documentation should improve over time. Finally, while most of the latest features in TensorFlow don't have the friendliest APIs right now, I'd be shocked if more user-friendly versions of TensorFlow Serving and the distributed runtime weren't in the works right now. Should I use TensorFlow? TensorFlow appears primed to fulfil the promise that was made back in November: a distributed, flexible data flow graph library that excels at neural network composition. I leave it to you decision makers to figure out whether TensorFlow is the right move for your own machine learning tasks, but here is my overall impression of TensorFlow: no other machine learning framework targeted at production-level tasks is as flexible, powerful, or improving as rapidly as TensorFlow. While other frameworks may carry advantages over TensorFlow now, Google is putting the effort into making consistent improvements, which bodes well for a community that is still in its infancy. About the author Sam Abrahams is a freelance data engineer and animator in Los Angeles, CA. He specializes in real-world applications of machine learning and is a contributor to TensorFlow. Sam runs a small tech blog, Memdump, and is an active member of the local hacker scene in West LA.