Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
All Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

Tech Guides

851 Articles
article-image-online-learning-new-classroom
Mark-Anthony Torrocha
04 Aug 2015
5 min read
Save for later

Online Learning: The New Classroom

Mark-Anthony Torrocha
04 Aug 2015
5 min read
Our Skill Up reports have clearly shown how learning new skills can increase your salary and what knowledge is most in demand in key areas of IT. So now you know what you need to learn, how do you actually go about learning it? Mark-Anthony Torrocha from coding school Coding Campus explains how online learning can be the best option if you are looking for flexibility, although you need to have commitment to make it work well. Simon Dudley notes that without a degree, you may not be able to earn as much over the course of your career. At some point it does seem to become economically unfeasible to earn that degree conventionally.  Most Coding Campus students, especially distance learners, come from promising careers before attending our bootcamp. Business owners, marketing directors, program managers all want the opportunity to create something of their own. They have spent time and money earning a bachelor's, sometimes even a master's degree in their profession and know that it is a risk to start over. For online students, it is even harder to think about starting a program that would take time away from their work and family. A great deal of thought and preparation must be taken, not just by the student, but also their families. When Breana Gonzales decided to take the Python Web Development course, she knew that it would take all her time to be a successful web developer. It was a conscious decision to take the course. The same goes for other online students, David Fall and Ervin Kleitz Gonzales, when they immersed themselves in the Angular Web Development course. Fall was working for a small company in Washington, while Gonzales did side work while telecommuting for the nighttime course. "You really need to be disciplined when you are taking the classes as an online student," remarked instructor Mark Price. "The success is in keeping up with the work and remembering that you are investing time and money into what you are doing." Issues Though Gonzales recognizes that a degree in Computer Science would give her greater income, she had bills to pay, and other things to do, but not the time to go the traditional route. Both Fall and Gonzales have the computer science background from their university education, but the rigorous training provided by the course gave them leverage in their new roles as web developers in their new found jobs. After completing the course and gaining experience at Verisage, Gonzales now works at EMC as an applications developer. Her course led her to her job, which allowed for her to fulfill her other plans. As much as time is an issue, money to participate in a course is also a key indicator of participation. A full-time immersive course can cost close to ten thousand dollars. Twelve weeks without a source of income can be daunting for many. Most students that take our online option choose a part-time course that allows for them to keep their current jobs while learning new skills. Some will take out loans to fund their time in the bootcamp. There are also those who choose to delay their programming education in hopes to getting enough money that they can afford the program. Unfortunately, we don't have access to federal money like most schools, but we do want to help in jumpstarting people's programming career. Here are other ways online bootcamps help with minimizing your costs: Climb Credit Coding Campus teams up with Climb Credit allowing students to finance their schooling Scholarship for Women, Minorities and Servicemen We offer 10 percent discounts, sometimes even 30 percent, to qualified individuals. Scholarships have been provided by civic employment programs, local companies, and others who really want you to succeed. Refer a friend and get even more money. On the Job training Companies have invested money in Coding Campus by means of projects that students work on during the course. Some even consider your time as a job, or apprenticeship, that you can refer to on your resume. It’s all about priorities "Despite discouraging headlines, a college degree remains the best route to finding a job in this tight market," TICAS president Lauren Asher said in a statement. Distance learners accepted by Coding Campus go through a stern admission process giving them the time to figure out if the course is something that they are able to commit to making them accountable to their learning. Before securing a job as a programmer, Bobby Clawson made a living as a firefighter. Turning to a career in programming was a step to provide for his growing family. “If you’re going to go the route of not going to college and not getting a degree for anything, then of course you’ll have to be prepared to find a lot of ways to study yourself which can be its own challenge,” Clawson said. “So I guess that has to be the type of thing that you have to weigh or consider for which path you want to go.” Coding Campus puts the same kind of attention to its distance learners as its in-person participants. Sometimes, they even require more. Distance learners are required to participate in real-time, turn in their assignments in local time, and maintain their attention to the course without physically being in the class. It is a lot of responsibility which is made known long before the course starts in order to prepare the students for the challenges. Because of these expectations, our distance learners thrive. Distance learning works for our students because it allows them time with their families, provides them with a real classroom experience, and an understanding that we provide the resources and it is up to them to apply it in their careers. To learn more about your options in starting a career in web development, visit us here.
Read more
  • 0
  • 0
  • 1094

article-image-biggest-app-developer-salary-and-skills-survey-2015
Packt Publishing
03 Aug 2015
1 min read
Save for later

The biggest App Developer salary and skills survey of 2015

Packt Publishing
03 Aug 2015
1 min read
See the highlights from our comprehensive Skill Up IT industry salary reports, with data from over 20,000 developers. What skills do you need to learn in order to earn more in application development?  Download the full size infographic here.    
Read more
  • 0
  • 0
  • 1438

article-image-biggest-big-data-and-business-intelligence-salary-and-skills-survey-2015
Packt Publishing
03 Aug 2015
1 min read
Save for later

The biggest Big Data & Business Intelligence salary and skills survey of 2015

Packt Publishing
03 Aug 2015
1 min read
See the highlights from our comprehensive Skill Up IT industry salary reports, with data from over 20,000 IT professionals. Find out what trends are emerging in the world of data science and business intelligence and what skills you should be learning to further your career. Download the full size infographic here.    
Read more
  • 0
  • 0
  • 1446

article-image-biggest-sysadmin-and-security-salary-and-skills-survey-2015
Packt Publishing
03 Aug 2015
1 min read
Save for later

The biggest Sysadmin and Security salary and skills survey of 2015

Packt Publishing
03 Aug 2015
1 min read
See the highlights from our comprehensive Skill Up IT industry salary reports, with data from over 20,000 IT professionals. Read on to discover which skills you should learn and which industry to get into to earn the big bucks! Download the full size infographic here.    
Read more
  • 0
  • 0
  • 2023

article-image-devops-engineering-and-full-stack-development
Richard Gall
28 Jul 2015
5 min read
Save for later

DevOps engineering and full-stack development – 2 sides of the same agile coin

Richard Gall
28 Jul 2015
5 min read
Two of the most talked-about and on-trend roles in tech dominated our Skill Up survey – DevOps engineers and Full-Stack developers. Even before we started exploring our data, we knew that both would feature heavily. Given the amount of time spent online arguing about DevOps and the merits and drawbacks of full-stack development, it’s interesting to see exactly what it means to be a DevOps engineer or full-stack developer. From salary to tool use, both our Web Development and SysAdmin and Security Salary and Skills Reports offer an insight into the professional lives of people actually performing these roles every day. The similarities between DevOps engineering and full-stack development The similarities between the two roles are striking. Both DevOps engineering and full-stack development are having a considerable impact on the way in which technology is used and understood within organizations and businesses – which makes them particularly valuable. In SMEs, for example, DevOps engineers command almost the same amount of money as in Enterprise. Considering the current economic climate, it’s a clear signal of the value of DevOps practices in environments where flexibility and the ability to adapt to changing demands and expectations are crucial to growth. Full-stack developers also command the highest salaries in many industries. In consultancy, for example, full-stack developers earn significantly more than any other web development role. While this could suggest that organizations aren’t yet willing to invest in (or simply don’t need) in-house full-stack developers, it highlights that they are nevertheless willing to spend money on individuals with full-stack knowledge, who are capable of delivering cutting-edge insight. However, just as we saw Cloud consultancies dominate the tech consultancy market a few years ago, over time it’s likely that full-stack development will become more and more established as a standard. DevOps engineers and full-stack developers share the same philosophical germ. They are symptoms of a growing business demand for greater agility and flexibility, and hint at a trend towards greater generalization in the skillset of technical professionals. part of the thrill of #devops to me is how there's no true agreement about what it is. it's like watching LOST all over again — jon devops hendren (@devops) May 18, 2015 Full-stack developers are using DevOps tools I’ve always seen them as manifestations of similar ideas in different technical areas. However, when you look at the data we’ve collected in our survey, alongside some wider research, the relationship between the DevOps engineer and the Full-Stack developer might possibly be more than purely conceptual. ‘Full-Stack’ and ‘DevOps’ are both terms that blur the lines between developer and engineer, and both are two sides of an intriguing form of cross-pollination; technologies more commonly used for deployment and automation. Docker and Vagrant were the most notable, highlighting the impact of containerization and virtualization on web development, but we also found a number of references to the Microsoft automation tool PowerShell – a distinctly DevOps-esque tool if ever there was one. Perhaps there’s a danger of overstating my point – surely we shouldn’t be surprised if web developers are using these tools – it’s not that strange, right? Maybe, but the fact that tools such as these are being used by web developers in their day-to-day work suggests that they are no longer simply expected to develop: they also need to deploy and configure their projects. Indeed, it’s worth noting that across all our web development respondents, a large number plan on learning Docker over the next 12 months. DevOps engineers use a huge range of tools DevOps Engineers were even more eclectic in their tool-usage than full-stack developers. Python is the language of-choice and Puppet the go-to configuration management tool, but web tools such as JavaScript and PHP are also being used. References to Flask, for example, the Python microframework, emphasise the way in which DevOps Engineers have an eye on web development while they’re automating your infrastructure. Taken alone, these responses might not fully evidence the relationship between DevOps engineers and Full-Stack developers. However, there are jobs out there asking for a combination of both skillsets. One, posted by a recruiter working for a nameless ‘creative media house’ in London, was looking for someone to become ‘a key member of multi-party cloud research projects, helping to bring a microservices-based video automation system to life, integrate development and developed systems into onside and global infrastructure’. The tools being asked for were very varied indeed. From a high-level language, such as JavaScript, to scripting languages such as Bash, Python and Perl, to continuous integration tools, configuration management tools and containerization technologies, whoever eventually gets the job certainly deserves to be called a polyglot. Blurring the line between full-stack and DevOps A further indication of the blurred line between engineers and developers can be found in this article from computing.co.uk. It’s an interesting tale of how working practices develop according to necessity and how methodologies and ideas interact with the practical details of a given situation. It tells the story of how the Washington Post went about building its submission platform, and how the way in which the project was resourced and managed changed according to certain pressures – internal and external. The title might actually be misleading – if you read it, it’s not so much that DevOps necessitates full-stack development, more that each thing grows out of the next. It might even be said that the reverse is true – that full-stack development necessitates DevOps thinking. The relationship between DevOps and full-stack development gives a real indication of the state of the tech world in 2015. Within a tech landscape of increasing complexity and cross-pollination there are going to be opportunities for developers and engineers to significantly drive their value as technical professionals. It’s simply a question of learning more, and of being open to new challenges and ideas about how to work effectively. It probably won’t be easy, but it might just be a fun journey.
Read more
  • 0
  • 0
  • 18230

article-image-what-are-skills-you-need-earn-more
Packt Publishing
27 Jul 2015
2 min read
Save for later

What are the skills you need to earn more?

Packt Publishing
27 Jul 2015
2 min read
In the 2015 Skill Up Survey, Packt talked to more than 20,000 people who work in IT globally to identify what skills are valued in technical roles and what trends are changing and emerging. Responses from the Data Science and Business Intelligence community provided us with some great insight into how skills are rated across multiple industries, job roles and experience levels. The world of Data Science is highly varied, and can be super competitive too, so we wanted to find out what industries are best for those just entering the market. We also discovered which technologies are proving to be most popular and where can you earn the best salaries. We also had some specific questions for which we wanted answers:- Are Python and R still the key data science languages? What are the hottest trends? Which is the most lucrative sector for those with Data Science skills. Which cutting edge technologies should you invest in learning? We received a whopping 3,800 responses from the Data Science community. The largest number of respondents were Consultants who, with over 20 years experience on average, provided some great insight into the Data Science world. Our survey identified which industry sectors pay the most for those with less experience and also showed up some big differences in average salaries across industries. We also wanted to look at the technologies our community are using. More than 25% of our respondents use Python, but is it here to stay, or will it be overtaken by Julia in the next 12 months? The results were interesting and suggested that salaries influence responses! Unsurprisingly, Big Data emerged as a Hot Topic. When asked whether their company was planning to implement a Big Data project over the next 12 months, it seems that the response was tied in to experience levels too. And of course, we just had to ask - does Excel still hold a place in your heart? Can you guess what the answer was? Read the rest of the report to see what skills you need to build on and which technologies are poised to take the Data Science and Business Intelligence world by storm so you can get ahead of the competition. Click here to download the full report Find out more about our exclusive Skill Up offers and discounts
Read more
  • 0
  • 0
  • 1085
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at €14.99/month. Cancel anytime
article-image-want-know-which-industry-sector-pays-25-more-it-administrators
Packt Publishing
27 Jul 2015
2 min read
Save for later

Want to know which industry sector pays 25% more for IT Administrators?

Packt Publishing
27 Jul 2015
2 min read
In the 2015 Skill Up Survey Packt talked to more than 20,000 people who work in IT globally to identify what skills are valued in technical roles and what trends are changing and emerging. Responses from the Security and System Administration community provided us with some great insight into how skills are rated across multiple industries, job roles and experience levels. The world of Security and System Admin constantly changes, is highly varied, and can be super competitive too, so we wanted to find out what industries are best for those just entering the market. We also discovered which technologies are proving to be most popular and where can you earn the best salaries. We also had some specific questions for which we wanted answers: What skills lead to a higher salary? What skills/technologies are most highly valued by different industries? What cutting edge technologies are really worth you spending your time learning? From our responses, for those with the least experience, Media/Advertising/Entertainment and Gaming industry sector is a great place to start! For those with more experience, Enterprise businesses are investing in Penetration Testers and Security Analysts, many of which are consulting, putting all those years of experience into practise. Take a look at your industry sector and see how your experience and salary compares with your peers. We also asked our respondents which tools they planned to use over the next 12 months to Skill Up. Malware Analysis topped the chart, but which other technologies are prominent, and who is learning what? Surprisingly, a number of non-technical skills, such as ‘coaching skills’, ‘change management’ and ‘project management’, were reported when asked what tools Administrators use, suggesting that roles are changing and softer skills are now part of an IT Administrator's role. Are you ready for that change? Read the rest of the report to see what technologies you need to build on, and which technologies are poised to take the Security and Sysadmin world by storm so you can get ahead of the competition. Click here to download the full report Find out more about our exclusive Skill Up offers and discounts
Read more
  • 0
  • 0
  • 1151

article-image-biggest-web-developer-salary-and-skills-survey-2015
Packt Publishing
27 Jul 2015
1 min read
Save for later

The biggest web developer salary and skills survey of 2015

Packt Publishing
27 Jul 2015
1 min read
The following infographic is taken from our comprehensive Skill Up IT industry salary reports, with data from over 20,000 developers. Download the full size infographic here.    
Read more
  • 0
  • 0
  • 1625

article-image-what-skill-sets-do-you-need-earn-more
Packt Publishing
24 Jul 2015
3 min read
Save for later

What skill sets do you need to earn more?

Packt Publishing
24 Jul 2015
3 min read
In the 2015 Skill Up Survey, Packt talked to more than 20,000 people who work in IT globally to identify what skills are valued in technical roles and what trends are changing and emerging. More than 6,000 of those worked in Web Development and provided us with some great insight into how skills are rated across multiple industries, job roles and experience levels. The world of web development constantly changes, and can be highly competitive, so we wanted to find out what industries are best for those just entering the market. We also discovered which technologies are proving to be most popular, where can you earn the best salaries and what skills you need to get that pay rise. We segmented our respondees into different types of developers and now know which developers are earning the highest average salaries, and which type of developers come second. We also know who earns the least, but at just under a $60k salary average, it’s still a great market to go into! If you’re about to move into your first role in Web Development our report can tell you which industry pays the most for those with less experience. Should you go for a role in Entertainment/Media, Government, Health or Finance? At the other end of the scale, those who have the longest experience tend to move into consulting, although the Government sector has its fair share of longer term experience, which is not all that surprising. Our survey results also gave an indication of where those in the middle of their career are employed. So what will be your next move? We wanted to test some assumptions as well. Such as: Is Angular 2.0 about to take over? How valuable is it to be a full stack developer? Are emerging economies taking a slice of the web pie? As expected, some of the results surprised us, and some confirmed that certain skills are becoming crucial to maintaining salary levels during your career. In terms of what technology skills you need to either enter the market or build on your experience, the top technologies are JavaScript, AngularJS and Python, so you’ll need to make sure you’re on top of these skill sets. Read the rest of the report to see what skills you need to build on and which technologies are poised to take web development by storm so you can get ahead of the competition. Click here to download the full report
Read more
  • 0
  • 0
  • 1341

article-image-do-you-have-technical-skills-command-90000-salary
Packt Publishing
24 Jul 2015
2 min read
Save for later

Do you have the technical skills to command a $90,000 salary?

Packt Publishing
24 Jul 2015
2 min read
In the 2015 Skill Up Survey Packt talked to more than 20,000 people who work in IT globally to identify what skills are valued in technical roles and what trends are changing and emerging. Responses from the App Development community provided us with some great insight into how skills are rated across multiple industries, job roles and experience levels. The world of App Development is highly varied, and can be super competitive too, so we wanted to find out what industries are best for those just entering the market. We also discovered which technologies are proving to be most popular and where can you earn the best salaries. We also had some very specific questions for which we wanted answers. How relevant are desktop developer skills? Which is the most popular platform for mobile development? Is functional programming the way of the future? What is the essential software choice for professional game development? Some of the results were surprising! Here’s a taster of our findings... If you are looking for your first role in App Development, the Government sector pays well for those with less experience. But competition is fierce - with under 5% of those working in this sector having less than 3 years experience. Unsurprisingly, Game Developers reported the lowest average salaries across all industries. It’s clear that Game Developers work for love, not money! However, which type of Developer earns the most and which industry sector is the most lucrative? Experienced Developers can find out who pays the most to utilize all of your expertise and experience. Also discover what Developers are building and what technologies they are using….here’s a sneak peek: And finally, what about the future? What is going to be the Next Big Thing? Wearables or Big Data perhaps? Is there a place for desktop in the mobile age? Read the rest of the report to see what skills you need to build on and which technologies are poised to take the App Development world by storm so you can get ahead of the competition! Click here to download the full report
Read more
  • 0
  • 0
  • 2698
article-image-making-it-games-development
Venita Pereira
16 Jul 2015
4 min read
Save for later

Making it in games development

Venita Pereira
16 Jul 2015
4 min read
Venita Pereira is a Senior Game Designer for Full Fat with over 8 years’ experience in the games industry, and the author of Learning Unity 2D Game Development by Example. She has worked for several AAA developers and on a vast range of genres and platforms. Recently, she worked on the Agent Dash update in Unity and was the sole designer for SEGA on the highly successful Sonic Dash also using the Unity Engine. She tweets as @Atinev and you can also find her on LinkedIn and Facebook. The big question – Unreal Engine versus Unity As the most widely used engine in the professional games industry, learning Unity is a great investment for a future career in games. Whilst some may argue that the Unreal Engine is the preferred choice (and I for one was previously an avid user of the Unreal Engine), Unity has caught up in leaps and bounds and many now consider it ahead of the curve. Skill Up Says… The Skill Up survey clearly shows that Unity is by far the stand-out technology for making professional-tier salaries in the game development industry. The highest paid game developers all know and regularly use the Unity engine. Many more games, especially indie and mobile games, use Unity; Unreal is better suited to AAA console games. That said, Unity has caught up – and even exceeded Unreal - with the introduction of physically based shaders. It is far more accessible and allows easy deployment on multiple platforms, so developers no longer need to work more to port their game. The most essential skills for the modern game developer Aside from technical knowledge, the essential skill of a games designer include an extensive understanding of game design principles, level design, and knowing how to design core game mechanics and features within a game design document. Communication and collaboration are also key, especially when you’re part of a professional team. Game developers need to understand revision control systems, data analytics, agile methodology and bug tracking. Knowledge of scripting and an understanding of the technical aspects of game development as well as the limitations are very beneficial. Thus, learning to use Unity, Word, Excel, Wiki, SVN or perforce and Confluence (or other tools with similar functions) are vital. The biggest challenges currently facing game developers The biggest challenge facing game developers today is retaining players in our games. This is nothing new! However, it is more challenging today due to the vast amount of games available. This makes user testing and marketing skills very valuable if you want to get ahead in the industry. The next big thing in game development The next big thing in game development will be smartwatches! Skill Up Says… One-in-two developers for mobile said on the Skill Up survey that they had plans to develop for the iWatch and other wearable devices. This is part of the continued moment for mobile gaming - the improvement in mobile hardware allows for more graphic and gameplay intensive games of console quality on mobile platforms. We’ll likely start to see a merging of consoles, PCs and mobiles. In addition, the growth and spread of Internet of Things means that we’re going to see improvements in networking, servers, cloud computing, and reactive systems. Programming languages like Rust and Go that are responsive, scalable and reliable are also going to mean big things for programming in general – and game development is no exception. My advice for anyone wanting to start out in game development or to take their career to the next level would be to keep learning and improving your skills! What is Skill Up? During June 2015, we surveyed over 20,000 IT professionals to find out what technologies they are currently using and planning to learn in the next 12 months. Download the report for free below to find out how you can Skill Up in games development. 
Read more
  • 0
  • 0
  • 1269

article-image-cloud-security-tips-locking-your-account-down-aws-identity-access-manger-iam
Robi Sen
15 Jul 2015
8 min read
Save for later

Cloud Security Tips: Locking Your Account Down with AWS Identity Access Manager (IAM)

Robi Sen
15 Jul 2015
8 min read
With the growth of cloud services such as Google’s Cloud Platform, Microsoft Azure, Amazon’s Web Service, and many others,developers and organizations have unprecedented access to low cost, high performance infrastructure that can scale as needed. Everyone from individuals to major companies have embraced the cloud as their platform of choice to host their IT services and applications; especially small companies and start-ups. Yet for many reasons, those who have embraced the cloud have often been slow to recognize the unique security considerations that face cloud users. When you host your own servers, the cloud operates on a shared risk model were the cloud provider focuses on providing physical security, failover, and high level network perimeter protection. The cloud user is understood to be securing their operating systems, data, applications, and the like. This means that while your cloud provider provides incredible services for your business, you are responsible for much of the security, including implementing access controls, intrusion prevention, intrusion detection, encryption, and the like. Often, because cloud services are made so accessible and easy to setup, users don’t bother to secure them, nor often even know the need to. If you’re new to the cloud and new to security, this post is for you. While we will focus on using Amazon Web Services,the basic concepts apply to most cloud services regardless of vendor. Access control Since you’re using virtual resources that are already setup, the AWS cloud, one of the most important things you need to do right away is secure access to your account and images. First, you want to lock down your AWS account. This is the login and password that you are assigned when you setup your AWS account and anyone who has access to this can purchase new services, change your services, and generally cause complete havoc for you. Indeed AWS accounts sell on hacker’s sites and Darknet sites for good money; usually so the person who buys your hacked/stolen AWS account wants to setup bit coin miners. Don’t give yours out or make it easily accessible. For example, many developers embed logins, passwords, and AWS keys into their code, which is a very bad practice, and then have their accounts compromised by criminals. The first thing you need to do after getting your Amazon login and password is to store it using a tool such as mSecure or LastPass that allows you to save them in an encrypted file or database. It should then never go into a file, document, or public place. It is also strongly advised to use Multi-Factor Authentication (MFA). Amazon allows you MFA via physical devices or straightforward smart phone applications. You can read more about Amazon’s MFA here and here. Once your AWS account information is secure you should then use AWS’s Identity and Access Management (IAM) system to give each user under your master AWS account access with specific privileges according to best practices. Even if you are the only person who uses your AWS account, you should consider using IAM to create a couple of users that have access based on their role, such as a content developer who only has the ability to move files in out of specific directories, or a developer who can start and stop instances, and the like. Then always use the role with the least privileges needed to get your work done as possible. While this might seem cumbersome, you will quickly get used to it, you will be much safer, and finally if your project grows, you will already have the groundwork to ramp up safely. Creating an IAM group and user In this section, we will create an administrator group and add ourselves as the user. If you do not currently have an AWS account, you can get a free account from AWS here. Be advised you will have to have a valid credit card and a phone number to validate your account with, but Amazon will give you the account to play with free for a year (see terms here). For this example, what you need to do is: Create an administrator group that we will give group permissions to for our AWS account’s resources Make a user for ourselves and then add the user to the administrator group Finally create a password for the user so we can access to the AWS Management Console To do this, first sign into the IAM console. Now click on the Groups link and then select Create New Group: Now name the new group Administrator and select Next Step: Next, we need to assign a group policy. You can build your own (see more information here), but this should generally be avoided until you really understand AWS security policies and AWS in general. Amazon has a number of predeveloped policy templates that work great until your applications and architecture gets more complex and grows. So for now just simply select the Administrator Access policy as shown here: You should now see a screen that shows your new policy. You can then click next and then Create Group: You should now see the new Administrator group policy under Group Name: In reality, you would probably want to create all your different group accounts and then associate your users, but for now we are just going to create the Administrator account then create a single user and add it to the Administrator group. Creating a new IAM user account Now that you have created an Administrator group, let's add a user to it. To do this, go to the navigation menu, select the user, and then click Create New Users. You should then see a new screen. You have the option to create access keys for this user. Depending on the user, you may or may not need to do this, but for now go ahead and select that option box and then select Create: IAM will now create the user and give you the option to view the new key or download and save it. Go ahead and download the credentials. Usually it’s good practice to then save those credentials into your password manager such as mSecure or LastPass and not share them with anyone except for the specific user. Once you have downloaded the userscredentials, go ahead and select Close, which will return you to the Users screen. Now click on the user you created. You should now see something like the following (the username has been removed from the figure): Now select Add User to Groups. You should now see the group listing, which only shows one if you’re following along.Now select the Administrator group and then select Add to Groups. You should be taken back to the Users content page and should now see that your user is assigned to the Administrator group. Now, staying in the same screen, scroll down until you see the Security Credentials part of the page. Now click Manage Password. You will now be asked to either select an auto-generated password or click Assign a custom password. Go ahead and create your own password and select Apply. You should be taken back to your user content screen and under security credentials section, you should now see that the password field has been updated from No to Yes. You should also strongly consider using your MFA tool, in my case the AWS virtual MFA Android application, to make the account even more secure. Summary In this article, we talked about the first step in securing your cloud services is controlling access to them. We looked at how AWS allows this via the IAM, allowing you to create groups and group security policies tied to a group, and then how to add users to the group enablingyou to secure your cloud resources based on best practices. Now that you have done that, you can go ahead and add more groups and or users to your AWS account as you need to.However, before you do that, make sure you thoroughly read the AWS IAM documentation; links are supplied at the end of the post. Resources for AWS IAM IAM User Guide Information on IAM Permissions and Policies IAM Best Practices About Author Robi Sen, CSO at Department 13, is an experienced inventor, serial entrepreneur, and futurist whose dynamic twenty-plus year career in technology, engineering, and research has led him to work on cutting edge projects for DARPA, TSWG, SOCOM, RRTO, NASA, DOE, and the DOD. Robi also has extensive experience in the commercial space, including the co-creation of several successful start-up companies. He has worked with companies such as UnderArmour, Sony, CISCO, IBM, and many others to help build out new products and services. Robi specializes in bringing his unique vision and thought process to difficult and complex problems allowing companies and organizations to find innovative solutions that they can rapidly operationalize or go to market with.            
Read more
  • 0
  • 0
  • 2794

article-image-reducing-cost-big-data-using-statistics-and-memory-technology-part-2
Praveen Rachabattuni
06 Jul 2015
6 min read
Save for later

Reducing Cost in Big Data using Statistics and In-memory Technology - Part 2

Praveen Rachabattuni
06 Jul 2015
6 min read
In the first part of this two-part blog series, we learned that using statistical algorithms gives us a 95 percent accuracy rate for big data analytics, is faster, and is a lot more beneficial than waiting for the exact results. We also took a look at a few algorithms along with a quick introduction to Spark. Now let’s take a look at two tools in depth that are used with statistical algorithms: Apache Spark and Apache Pig. Apache Spark Apache Spark is a fast and general-purpose cluster computing system. It provides high-level APIs in Java, Scala, and Python, as well as an optimized engine that supports general execution graphs. It also supports a rich set of higher-level tools including Spark SQL for SQL and structured data processing, MLlib for machine learning, GraphX for graph processing, and Spark Streaming. At its core, Spark provides a general programming model that enables developers to write applications by composing arbitrary operators, such as mappers, reducers, joins, group-bys, and filters. This composition makes it easy to express a wide array of computations, including iterative machine learning, streaming, complex queries, and batch processing. In addition, Spark keeps track of the data that each of the operators produces, and enables applications to reliably store this data in memory. This is the key to Spark’s performance, as it allows applications to avoid costly disk accesses. It would be wonderful to have one tool for everyone, and one architecture and language for investigative as well as operational analytics. Spark’s ease of use comes from its general programming model, which does not constrain users to structure their applications into a bunch of map and reduce operations. Spark’s parallel programs look very much like sequential programs, which make them easier to develop and reason about. Finally, Spark allows users to easily combine batch, interactive, and streaming jobs in the same application. As a result, a Spark job can be up to 100 times faster and requires writing 210 times less code than an equivalent Hadoop job. Spark allows users and applications to explicitly cache a dataset by calling the cache() operation. This means that your applications can now access data from RAM instead of disk, which can dramatically improve the performance of iterative algorithms that access the same dataset repeatedly. This use case covers an important class of applications, as all machine learning and graph algorithms are iterative in nature. When constructing a complex pipeline of MapReduce jobs, the task of correctly parallelizing the sequence of jobs is left to you. Thus, a scheduler tool such as Apache Oozie is often required to carefully construct this sequence. With Spark, a whole series of individual tasks is expressed as a single program flow that is lazily evaluated so that the system has a complete picture of the execution graph. This approach allows the core scheduler to correctly map the dependencies across different stages in the application, and automatically parallelize the flow of operators without user intervention. With a low-latency data analysis system at your disposal, it’s natural to extend the engine towards processing live data streams. Spark has an API for working with streams, providing exactly-once semantics and full recovery of stateful operators. It also has the distinct advantage of giving you the same Spark APIs to process your streams, including reuse of your regular Spark application code. Pig on Spark Pig on Spark combines the power and simplicity of Apache Pig on Apache Spark, making existing ETL pipelines 100 times faster than before. We do that via a unique mix of our operator toolkit, called DataDoctor, and Spark. The following are the primary goals for the project: Make data processing more powerful Make data processing more simple Make data processing 100 times faster than before DataDoctor is a high-level operator DSL on top of Spark. It has frameworks for no-symmetrical joins, sorting, grouping, and embedding native Spark functions. It hides a lot of complexity and makes it simple to implement data operators used in applications like Pig and Apache Hive on Spark. Pig operates in a similar manner to big data applications like Hive and Cascading. It has a query language quite akin to SQL that allows analysts and developers to design and write data flows. The query language is translated in to a “logical plan” that is further translated in to a “physical plan” containing operators. Those operators are then run on the designated execution engine (MapReduce, Apache Tez, and now Spark). There are a whole bunch of details around tracking progress, handling errors, and so on that I will skip here. Query planning on Spark will vary significantly from MapReduce, as Spark handles data wrangling in a much more optimized way. Further query planning can benefit greatly from ongoing effort on Catalyst inside Spark. At this moment, we have simply introduced a SparkPlanner that will undertake the conversion from a logical to a physical plan for Pig. Databricks is working actively to enable Catalyst to handle much of the operator optimizations that will plug into SparkPlanner in the near future. Longer term, we plan to rely on Spark itself for logical plan generation. An early version of this integration has been prototyped in partnership with Databricks. Pig Core hands off Spark execution to SparkLauncher with the physical plan. SparkLauncher creates a SparkContext providing all the Pig dependency JAR files and Pig itself. SparkLauncher gets an MR plan object created from the physical plan. At this point, we override all the Pig operators to DataDoctor operators recursively in the whole plan. Two iterations are performed over the plan — one that looks at the store operations and recursively travels down the execution tree, and a second iteration that does a breadth-first traversal over the plan and calls convert on each of the operators. The base class of converters in DataDoctor is a POConverter class and defines the abstract method convert, which is called during plan execution. More details of Pig on Spark can be found at PIG4059. As we merge with Apache Pig, we need to focus on the following enhancements to further improve the speed of Pig: Cache operator: Adding a new operator to explicitly tell Spark to cache certain datasets for faster execution Storage hints: Allowing the user to specify the storage location of datasets in Spark for better control of memory YARN and Mesos support: Adding resource manager support for more global deployment and support Conclusion In many large-scale data applications, statistical perspectives provide us with fruitful analytics in many ways, including speed and efficiency. About the author Praveen Rachabattuni is a tech lead at Sigmoid Analytics, a company that provides a real-time streaming and ETL framework on Apache Spark. Praveen is also a committer to Apache Pig.
Read more
  • 0
  • 0
  • 1088
article-image-reducing-cost-big-data-using-statistics-and-memory-technology-part-1
Praveen Rachabattuni
03 Jul 2015
4 min read
Save for later

Reducing Cost in Big Data using Statistics and In-memory Technology - Part 1

Praveen Rachabattuni
03 Jul 2015
4 min read
The world is shifting from private, dedicated data centers to on-demand computing in the cloud. This shift moves the onus of cost from the hands of IT companies to the hands of developers. As your data sizes start to rise, the computing cost grows linearly with it. We have found that using statistical algorithms gives us a 95 percent accuracy rate, is faster, and is a lot more beneficial than waiting for the exact results. The following are some common analytical queries that we have often come across in applications: How many distinct elements are in the data set (that is, what is the cardinality of the data set)? What are the most frequent elements (that is, the “heavy hitters” and “top elements”)? What are the frequencies of the most frequent elements? Does the data set contain a particular element (search query)? Can you filter data based upon a category? Statistical algorithms for quicker analytics Frequently, statistical algorithms avoid storing the original data, replacing it with hashes that eliminate a lot of network. Let’s get into the details of some of these algorithms that can help answer queries similar to those mentioned previously. A Bloom filter is a data structure designed to tell you, rapidly and memory-efficiently, whether an element is present in a set. It is suitable in cases when we need to quickly filter items that are present in a set. HyperLogLog is an approximate technique for computing the number of distinct entries in a set (cardinality). It does this while using only a small amount of memory. For instance, to achieve 99 percent accuracy, it needs only 16 KB. In cases when we need to count distinct elements in a dataset spread across a Hadoop cluster, we could compute the hashes on different machines, build the bit index, and combine the bit index to compute the overall distinct elements. This eliminates the need of moving the data across the network and thus saves us a lot of time. The Count–min sketch is a probabilistic sub-linear space streaming algorithm that can be used to summarize a data stream to obtain the frequency of elements. It allocates a fixed amount of space to store count information, which does not vary over time even as more and more counts are updated. Nevertheless, it is able to provide useful estimated counts, because the accuracy scales with the total sum of all the counts stored. Spark - a faster execution engine Spark is a faster execution engine that provides 10 times the performance over MapReduce when combined with these statistical algorithms. Using Spark with statistical algorithms gives us a huge benefit both in terms of cost and time savings. Spark gets most of its speed by constructing Directed Acyclic Graphs (DAGs) out of the job operations and uses memory to save intermediate data, thus making the reads faster. When using statistical algorithms, saving the hashes in memory makes the algorithms work much faster. Case study Let’s say we have a continuous stream of user log data coming every hour at a rate of 4.4 GB per hour, and we need to analyze the distinct IPs in the logs on a daily basis. At my old company, when MapReduce was used to process the data, it was taking about 6 hours to process one day’s worth of data at a size of 106 GB. We had an AWS cluster consisting of 50 spot instances and 4 on-demand instances running to perform the analysis at a cost of $150 per day. Our system was then shifted to use Spark and HyperLogLog. This shift brought down the cost to $16.50 per day. To summarize, we had a 3.1 TB stream of data processed every month at a cost of $495, which was costing about $4,500 on the original system using MapReduce without the statistical algorithm in place. Further reading In the second part of this two-part blog series, we will discuss two tools in depth: Apache Spark and Apache Pig. We will take a look at how Pig combined with Spark makes existing ETL pipelines 100 times faster, and we will further our understanding of how statistical perspectives positively effect data analytics. About the author Praveen Rachabattuni is a tech lead at Sigmoid Analytics, a company that provides a real-time streaming and ETL framework on Apache Spark. Praveen is also a committer to Apache Pig.
Read more
  • 0
  • 0
  • 1255

article-image-today-you-are-not-web-developer-if-you-dont-know-javascript
Mario Casciaro
01 Jul 2015
6 min read
Save for later

You're not a web developer if you don't know JavaScript

Mario Casciaro
01 Jul 2015
6 min read
Mario Casciaro is a software engineer and technical lead with a passion for open source. After the recent publication of his successful book Node.JS Design Patterns, we caught up with him to discuss his views on today’s most important web development skills, and what the future holds. The best tool for the job may not be in your skillset yet I remember working on a small side project, something I try to do as much as possible, to put new skills into practice and try things outside of my job. It was a web application, something very similar to a social network, and I remember choosing Java with the Spring Framework as the main technology stack, and Backbone on the front-end. At the time - around 4 years ago - I was an expert Java developer, and considered it the technology with the most potential. It worked perfectly to implement enterprise web applications as well as mission-critical distributed applications and even mobile apps. While Java is still a popular and valuable tool in 2015, my experience doing this small side project made me rethink my opinion – I wouldn’t use it today unless there was a particular need for it. I remember that at some point I realized I was spending a lot of my development time in designing the object-oriented structure of the application and writing boilerplate code. Trying to find a solution, I migrated the project to Groovy and Grails and on the front-end I tried to implement a small homemade two-way binding framework. Things improved a little, but I was still feeling the need for something more agile on both ends, something more suited to the web. The web moves fast, so always let your skills evolve I wanted to try something radically different than the typical PHP, Ruby on Rails or Python for the server or JQuery or Backbone for the client. Fortunately I discovered Node.js and Angular.js, and that changed everything. By using Node I noticed that my mindset shifted from “how to do things” to “just get things done”. On the other hand, Angular revolutionized my approach to front end development, allowing me to drastically cut the amount of boilerplate code I was using. But most importantly, I realized that JavaScript and its ecosystem was becoming a seriously big thing. Today I would not even consider building a web application without having JavaScript as my primary player. The amount of packages on npm is staggering - a clear indication that the web has shifted towards JavaScript. The most impressive part of this story is that I also realized the importance that these new skills had in defining my career; if I wanted to build web applications, JavaScript and its amazing ecosystem had to be the focus of my learning efforts. This led me to find a job where Node, Angular and other cutting-edge JavaScript technologies actually played a crucial role in the success of the project I was in charge of creating. But the culmination of my renewed interest in JavaScript is the book I published 6 months ago - Node.jsDesignPatterns - which contains the best of the experience I accumulated since I devoted myself to the full-stack JavaScript mantra. The technologies and the skills that matter today for a web developer Today, if I had to give advice to someone approaching web development for the first time I would definitely recommend starting with JavaScript. I wouldn’t have said that 5-6 years ago, but today it’s the only language that allows you to get started both on the back end and the front end. Moreover JavaScript, in combination with other web technologies such as HTML and CSS, gives you access to an even broader set of applications with the help of projects like nw.js and ApacheCordova. PHP, Ruby, and Python are still very popular languages for developing the server-side of a web application, but for someone that already knows JavaScript, Node.js would be a natural choice. Not only does it save you the time it takes to learn a new language, it also offers a level of integration with the front end that is impossible with other platforms. I’m talking, of course, about sharing code between the server and the client and even implementing isomorphic applications which can run on both Node.js and the browser. React is probably the framework that offers the most interesting features in the area of isomorphic application development and definitely something worth digging into more, and it’s likely that we’ll also see a lot more from PouchDB, an isomorphic JavaScript database that will help developers build offline-enabled or even offline-first web applications more easily than ever before. Always stay ahead of the curve Today, as 4 years ago, the technologies that will play an important role in the web of tomorrow are already making an impact. WebRTC, for example, enables the creation of real-time peer-to-peer applications in the browser, without the need for any additional plugin. Developers are already using it to build fast and lightweight audio/video conferencing applications or even complete BitTorrent clients in the browser! Another revolutionizing technology is going to be ServiceWorkers which should dramatically improve the capabilities of offline applications. On the front end, WebComponents are going to play a huge role, and the Polymer project has already demonstrated what this new set of standards will be able to create. With regards to JavaScript itself, web developers will have to become familiar with the ES6 standard sooner than expected, as cross-compilation tools such as Babel are already allowing us to use ES6 on almost any platform. But we should also keep an eye on ES7 as it will contain very useful features to simplify asynchronous programming. Finally, as the browser becomes the runtime environment of the future, the recently revealed WebAssembly promises to give the web its own “bytecode”, allowing you to load code written in other languages from JavaScript, When WebAssembly becomes widely available, it will be common to see a complex 3D video game or a full-featured video editing tool running in the browser. JavaScript will probably remain the mainstream language for the web, but it will be complemented by the new possibilities introduced by WebAssembly. If you want to explore the JavaScript ecosystem in detail start with our dedicated JavaScript page. You'll find our latest and most popular, along with free tutorials and insights.
Read more
  • 0
  • 1
  • 3031