Search icon CANCEL
Subscription
0
Cart icon
Your Cart (0 item)
Close icon
You have no products in your basket yet
Save more on your purchases! discount-offer-chevron-icon
Savings automatically calculated. No voucher code required.
Arrow left icon
All Products
Best Sellers
New Releases
Books
Videos
Audiobooks
Learning Hub
Newsletter Hub
Free Learning
Arrow right icon
timer SALE ENDS IN
0 Days
:
00 Hours
:
00 Minutes
:
00 Seconds

How-To Tutorials

6719 Articles
article-image-mission-running-eve-online
Packt
17 Oct 2012
14 min read
Save for later

Mission Running in EVE Online

Packt
17 Oct 2012
14 min read
Mission types Missions in EVE fall into five general categories: Courier, Kill, Mining, Trade, and Special missions. Courier, Kill, Mining, and Trade missions are further categorized into five different levels: level I through level V. Let's take a closer look at each category to see what each type of mission entails. Courier missions Courier missions are simply delivery missions where you move items from one station to another. The type of items can range from common items found on the market to mission-specific cargo containers. The size and quantity of items to move can vary greatly and can either be moved using whatever ship you are currently flying or may require the use of an industrial ship.   While the ISK reward for courier missions is a bit on the low side, it having no negative standing impact on opposing factions is a huge positive. This, added to the fact that courier missions are quite easy and can often be completed very quickly, means they are generally worth running. Every so often you will receive a courier mission that takes you into Low-Sec space and with the risks involved in Low-Sec, it is best to decline these missions. You are able to decline one mission every four hours per agent without taking a negative standing hit with that agent. Be very careful when declining missions, since if your standing falls too low with an agent you will no longer be able to receive mission offers from that agent. Kill missions By far the most common, the most profitable, and let's be honest, the most fun missions to run are kill missions. The only thing better than flying beautiful space ships is seeing beautiful space ships blow up. There are currently two types of kill missions. In one, you warp to a set of coordinates in space where you engage NPC targets and in the other you warp to an acceleration gate which takes you into a series of pockets all connected by more acceleration gates. Think of this second type of kill mission like a dungeon with multiple rooms, in which you fight NPC targets in each of the rooms. Kill missions are a great way to make ISK, especially when you are able to access higher-level agents. However, as you can guess, kill missions also get more and more difficult the higher the level of the agent. Another thing that you need to be very careful about when running kill missions is that you run the risk of having negative standing impact on factions opposed to the faction you are running missions for. For example, if you were running missions for the Caldari state and you completed a mission that had you destroy ships belonging to or friendly with the Gallente Federation, then you would lose standing with the Gallente. A great way to negate the standing loss is to run missions for as many agents as you can find and to decline any mission that will have you attack the ships of another empire. But remember you can only decline one mission per agent every four hours, hence having multiple agents. Agents and how they work Now would be a great time to take a closer look at the agent system of EVE. Each agent works in a specific division of a NPC corporation. The division that the agent works in determines the type of missions you will receive (more on this later), and the NPC corporation that the agent works for determines which faction your standing rewards will affect. An agent's information sheet is shown in the following image: As you can see, the agent Aariwa Tenanochi works for the Home Guard in their Security division and is a level 1 agent. Aariwa can be found in the Nonni system at the Home Guard Assembly Plant located at planet III Moon 1. You can also see that the only requirement to receive missions from Aariwa is that you meet his standing requirements. Doing the tutorial missions will allow you to access all level 1 agents for your chosen faction. All agents are rated level 1 to level 5. The higher the level of the agent the harder their missions are, and the harder the mission the better and bigger the reward. The difficulty level of each agent level can best be described as follows: Level 1: Very Easy. Designed for new players to learn about mission running and PvE in general. Frigate or destroyer for kill missions, short range and low cost items for courier and trade missions, and small amounts of low grade ore or minerals for mining missions. Level 2: Easy. Still designed beginner players, but will require you to start thinking tactically. Destroyer or cruiser for kill missions, slightly longer range and higher cost items for courier and trade missions, and higher amounts of low grade ore or minerals for mining missions. Level 3: Medium. You will need to have a firm understanding of how your ship works and have solid tactics for it. Battlecruiser or Heavy Assault Cruiser for kill missions, much longer range and higher cost items for courier and trade missions, and large amounts of low grade ore or minerals for mining missions. Level 4: Hard. You will need to understand your ship inside out, have solid tactics, and a solid fit for your ship. Battleship for kill missions, very long range and very costly items for courier and trade missions, and very large amounts of low grade ore or minerals for mining missions. Level 5: Very Hard. Same as level 4 agents but only found in Low-Sec systems and comes with all the risks of Low-Sec. It is a good idea to only do these missions in a fleet with a group of people. The standing requirements to access the different level of agents are shown in the following table:   Level 1 Level 2 Level 3 Level 4 Level 5 Standing required - 1.00 3.00 5.00 7.00 While you can use larger ships to run lower level kill missions, it is generally not a good idea. First, there can be ship limitation for a mission that does not allow you to use your larger ship and second, while your larger ship will be able to handle the incoming damage much easier, the larger weapons that your ship uses will have a harder time hitting the smaller targets. And since ammo costs ISK, it cuts into your profit per mission. I generally like to use the cheapest ship I can get away with when running missions. This ensures that if I ever get scanned down and killed by player pirates, my loss is minimal. Understanding your foe The biggest key to success when running kill missions is understanding who your targets will be and how they like to fight. Whenever you are offered a kill mission by an agent, you can find out which faction your target belongs to inside the mission description, either in the text of the description or by an emblem representing your target's faction. It is important to know which faction your target belongs to because then you will know what kind of damage to expect from your target. In the mission offer shown in the following image you can see that your targets for this mission are Rogue Drones: There are four types of damage, EM, Thermal , Kinetic, and Explosive, and each faction has a preference toward specific damage types. Once you know what kind of damage to expect you can then tailor the defense of your ship for maximum protection towards that damage. Knowing which faction your target belongs to will also tell you which type of damage you should be doing to maximize your damage output. Each faction has weaknesses towards specific damage types. Along with what types of damage to defend against and what types of damage to utilize, knowing which faction your target belongs to will also tell you what kind of special tactic ships you may encounter in your mission. For example, the Caldari likes to use ECM to jam your targeting systems and the Amarr uses energy neutralizers to drain your ship's capacitor, leaving you unable to use weapons or even to warp away. The following table shows all the factions and the damage types to defend and use: NPC faction Damage type to defend Damage type to deal Guristas Kinetic, Thermal Kinetic, Thermal Serpentis Thermal, Kinetic Thermal Blood Raider EM, Thermal EM, Thermal Sansha's Nation EM, Thermal EM, Thermal Angel Cartel Explosive, Kinetic, Thermal, EM Explosive Mordu's Legion Kinetic, Thermal, Explosive, EM Thermal, Kinetic Mercenary Kinetic, Thermal Thermal, Kinetic Republic Fleet Explosive, Thermal, Kinetic, EM Explosive, Kinetic Caldari Navy Kinetic, Thermal Kinetic, Thermal Amarr Navy EM, Thermal, Kinetic EM, Thermal Federation Navy Thermal, Kinetic Kinetic, Thermal Rogue Drones Explosive, Kinetic, EM, Thermal EM Thukker Tribe Explosive, Thermal EM CONCORD EM, Thermal, Kinetic, Explosive Explosive, Kinetic Equilibrium of Mankind Kinetic, Thermal Kinetic, EM You will most likely have noticed that several factions use the same damage types but they are listed in different orders. The damage types are listed this way because the damage output or weakness is not divided evenly. So for the Serpentis pirate faction for example, they prefer a combination of Kinetic and Thermal damage with a higher percentage being Kinetic damage than Thermal damage. What ship to use to maximize earnings The first question that most new mission runners ask is "What race of ships should I use?" While each race has its own advantages and disadvantages and are all very good choices, in my opinion Gallente drone ships are the best ships to use for mission running, from a pure ISK making stand point. Gallente drone ships are the best when it comes to mission running because of the lack of ammo costs and the versatility they offer. With drones being your primary method of dealing damage you do not have to worry about the cost associated with needing a large supply of ammo like the Caldari or Minmatar. What about the Amarr you say? They don't need ammo. While it is true that the Amarr also do not require ammo, the heavy capacitor drain of lasers limits the amount of defense your ship can have. Drone ships do not need to fit weapons in their high slots and therefore you can fit your ship with the maximum amount of defense possible. Drone ships can also utilize the speed and range of drones to their advantage. By using modules such as the Drone Link Augmentor you can increase the range of your drones so that you can comfortably sit outside the attack range of your targets and let your drones do all the work. Being outside of the attack range also means that you are outside the range of electronic warfare, so it's a win-win situation. The best feature of drone ships is the ability to carry different types of drones. Light Drones for frigate-sized targets, Medium Drones for cruiser-sized targets, and Heavy Drones for battleship-sized targets. You can even carry Electronic Warfare Drones that can jam your targets, drain their capacitors, or even provide more defense for your ship by way of shield or armor repairs. How should I fit my ship? The second most common question is "How should I have my ship fitted for mission running?" Ship fitting is very subjective and almost an art form in itself. I can easily go on for pages about the different concepts behind ship fittings and why one way is better than another but there will always be people that will disagree because what works for one person does not necessarily work for you. The best thing to do is to visit websites such as eve.battleclinic.com/browse_loadouts.php to get ideas on how ship fittings should work and to use 3rd party software such as Python Fitting Assistant to come up with fittings that suit your style of play the best. When browsing fittings on BattleClinic, it is a good idea to make sure the filters at the bottom right are set for the most recent expansions. You wouldn't want to use a completely out-of-date fitting that will only get you killed. Basic tactics When you start a kill mission you will likely be faced with two different scenarios. The first is as soon as you come out of warp, targets will be on the offensive and immediately start attacking you. The second and more favorable scenario is to warp onto the field with multiple groups of targets in a semi-passive state. No matter the scenario you come across, the following are a few simple tactics will ensure everything goes that much smoother. These tactics are as follows: Zoom out. I know it's hard because your ship is so pretty, but zoom the camera out. You need to be able to have a high level of situational awareness at all times and you can't do that if you're zoomed in on your ship all the time. Finish each wave or group of targets before engaging other targets. This will ensure that you will never be too outnumbered and be able to more easily handle the incoming damage. Kill the tacklers first. Tacklers are ships that use electronic warfare to slow your ship or to prevent you from warping away. Since flying out of range or warping away is the best way of escape when things go bad, killing off the tacklers first will give you the best chance to escape intact. Kill the largest targets first. Taking out the largest targets first gives you two critical advantages. The first is you are taking a lot of the damage against you off the field and the second is that larger ships are much easier to hit. Save structures for last. If part of your mission is to shoot or destroy a structure, save that for last. The majority of times, firing on a structure will cause all the hostiles to attack you at once and may even spawn stationary defenses. This tactic does not apply to defensive towers, such as missile and tackling towers. You should always kill these towers as soon as possible. Mining missions Mining missions come in two flavors. The first will have you travel to a set of coordinates given to you by your agent, to mine a specific amount of a specific type of ore and then return to your agent with the ore you have mined. The second will simply have you supply your agent with an amount of ore or mineral. For the second type of mining mission you can either mine and refine the ore yourself or purchase it from the market. The ISK reward for mining missions is really bad and in general you will make more ISK if you had simply spent the time mining, but once again, having no negative standing impact on opposing factions is a huge plus. So if you are a career miner, it can be worth it for you to run these missions for the standings gain. After all, you will have to increase your standing in order to maximize your refine yield. It would be best to only accept the missions in which you already have the requested ore or mineral in your stock and to decline the rest. Trade missions Trade missions are simple missions that require you to provide items to your agent at a specific station. These items can either be made by you or purchased off the market. Like courier and mining missions, the only upside to trade missions is that they can be completed without the negative standing impact on opposing factions. But with the high amount of ISK needed to complete these missions and the time involved, it is best to avoid these missions. If you choose to do these missions, check to see if the item required is on sale at the destination station. If it is on sale there, you can often purchase it and complete the mission without ever leaving your current station.
Read more
  • 0
  • 0
  • 8440

article-image-how-to-build-a-basic-server-side-chatbot-using-go
Sunith Shetty
19 Apr 2018
20 min read
Save for later

How to build a basic server side chatbot using Go

Sunith Shetty
19 Apr 2018
20 min read
It's common nowadays to see chatbots (also known as agents) service the needs of website users for a wide variety of purposes, from deciding what shoes to purchase to providing tips on what stocks would look good on a client's portfolio. In a real-world scenario, this functionality would be an attractive proposition for both product sales and technical support usage scenarios. For instance, if a user has a particular question on a product listed on the website, they can freely browse through the website and have a live conversation with the agent. In today’s tutorial, we will examine the functionality required to implement the live chat feature on the server side chatbot. Let’s look at how to implement a live chat feature on various product related pages. In order to have the chat box present in all sections of the website, we will need to place the chat box div container right below the primary content div container in the web page layout template (layouts/webpage_layout.tmpl): <!doctype html> <html> {{ template "partials/header_partial" . }} <div id="primaryContent" class="pageContent"> {{ template "pagecontent" . }} </div> <div id="chatboxContainer" class="containerPulse"> </div> {{ template "partials/footer_partial" . }} </html> The chat box will be implemented as a partial template in the chatbox_partial.tmpl source file in the shared/templates/partials folder: <div id="chatbox"> <div id="chatboxHeaderBar" class="chatboxHeader"> <div id="chatboxTitle" class="chatboxHeaderTitle"><span>Chat with {{.AgentName}}</span></div> <div id="chatboxCloseControl">X</div> </div> <div class="chatboxAgentInfo"> <div class="chatboxAgentThumbnail"><img src="{{.AgentThumbImagePath}}" height="81px"></div> <div class="chatboxAgentName">{{.AgentName}}</div> <div class="chatboxAgentTitle">{{.AgentTitle}}</div> </div> <div id="chatboxConversationContainer"> </div> <div id="chatboxMsgInputContainer"> <input type="text" id="chatboxInputField" placeholder="Type your message here..."> </input> </div> <div class="chatboxFooter"> <a href="http://www.isomorphicgo.org" target="_blank">Powered by Isomorphic Go</a> </div> </div> This is the HTML markup required to implement the wireframe design of the live chat box. Note that the input textfield having the id "chatboxInputField". This is the input field where the user will be able to type their message. Each message created, both the one that the user writes, as well as the one that the bot writes, will use the livechatmsg_partial.tmpl template: <div class="chatboxMessage"> <div class="chatSenderName">{{.Name}}</div> <div class="chatSenderMsg">{{.Message}}</div> </div> Each message is inside its own div container that has two div containers (shown in bold) housing the name of the sender of the message and the message itself. There are no buttons necessary in the live chat feature, since we will be adding an event listener to listen for the press of the Enter key to submit the user's message to the server over the WebSocket connection. Now that we've implemented the HTML markup that will be used to render the chat box, let's examine the functionality required to implement the live chat feature on the server side. Live chat's server-side functionality When the live chat feature is activated, we will create a persistent, WebSocket connection, between the web client and the web server. The Gorilla Web Toolkit provides an excellent implementation of the WebSocket protocol in their websocket package. To fetch the websocket package, you may issue the following command: $ go get github.com/gorilla/websocket The Gorilla web toolkit also provides a useful example web chat application. Rather than reinventing the wheel, we will repurpose Gorilla's example web chat application to fulfill the live chat feature. The source files needed from the web chat example have been copied over to the chat folder. There are three major changes we need to make to realize the live chat feature using the example chat application provided by Gorilla: Replies from the chatbot (the agent) should be targeted to a specific user, and not be sent out to every connected user We need to create the functionality to allow the chatbot to send a message back to the user We need to implement the front-end portion of the chat application in Go Let's consider each of these three points in more detail. First, Gorilla's web chat example is a free-for-all chat room. Any user can come in, type a message, and all other users connected to the chat server will be able to see the message. A major requirement for the live chat feature is that each conversation between the chatbot and the human should be exclusive. Replies from the agent must be targeted to a specific user, and not to all connected users. Second, the example web chat application from the Gorilla web toolkit doesn't send any messages back to the user. This is where the custom chatbot comes into the picture. The agent will communicate directly with the user over the established WebSocket connection. Third, the front-end portion of the example web chat application is implemented as a HTML document containing inline CSS and JavaScript. As you may have guessed already, we will implement the front-end portion for the live chat feature in Go, and the code will reside in the client/chat folder. Now that we have established our plan of action to implement the live chat feature using the Gorilla web chat example as a foundation to start from, let's begin the implementation. The modified web chat application that we will create contains two main types: Hub and Client. The hub type The chat hub is responsible for maintaining a list of client connections and directing the chatbot to broadcast a message to the relevant client. For example, if Alice asked the question "What is Isomorphic Go?", the answer from the chatbot should go to Alice and not to Bob (who may not have even asked a question yet). Here's what the Hub struct looks like: type Hub struct {  chatbot bot.Bot  clients map[*Client]bool  broadcastmsg chan *ClientMessage register chan *Client  unregister chan *Client } The chatbot is a chat bot (agent) that implements the Bot interface. This is the brain that will answer the questions received from clients. The clients map is used to register clients. The key-value pair stored in the map consists of the key, a pointer to a Client instance, and the value consists of a Boolean value set to true to indicate that the client is connected. Clients communicate with the hub over the broadcastmsg, register, and unregister channels. The register channel registers a client with the hub. The unregister channel, unregisters a client with the hub. The client sends the message entered by the user over the broadcastmsg channel, a channel of type ClientMessage. Here's the ClientMessage struct that we have introduced: type ClientMessage struct {  client *Client  message []byte } To fulfill the first major change we laid out previously, that is, the exclusivity of the conversation between the agent and the user, we use the ClientMessage struct to store, both the pointer to the Client instance that sent the user's message along with the user's message itself (a byte slice). The constructor function, NewHub, takes in chatbot that implements the Bot interface and returns a new Hub instance: func NewHub(chatbot bot.Bot) *Hub {  return &Hub{    chatbot: chatbot,    broadcastmsg: make(chan *ClientMessage), register: make(chan    *Client), unregister:        make(chan *Client),    clients: make(map[*Client]bool),  } } We implement an exported getter method, ChatBot, so that the chatbot can be accessed from the Hub object: func (h *Hub) ChatBot() bot.Bot {  return h.chatbot } This action will be significant when we implement a Rest API endpoint to send the bot's details (its name, title, and avatar image) to the client. The SendMessage method is responsible for broadcasting a message to a particular client: func (h *Hub) SendMessage(client *Client, message []byte) {  client.send <- message } The method takes in a pointer to Client, and the message, which is a byte slice, that should be sent to that particular client. The message will be sent over the client's send channel. The Run method is called to start the chat hub: func (h *Hub) Run() { for { select { case client := <-h.register: h.clients[client] = true greeting := h.chatbot.Greeting() h.SendMessage(client, []byte(greeting)) case client := <-h.unregister: if _, ok := h.clients[client]; ok { delete(h.clients, client) close(client.send) } case clientmsg := <-h.broadcastmsg: client := clientmsg.client reply := h.chatbot.Reply(string(clientmsg.message)) h.SendMessage(client, []byte(reply)) } } } We use the select statement inside the for loop to wait on multiple client operations. In the case that a pointer to a Client comes in over the hub's register channel, the hub will register the new client by adding the client pointer (as the key) to the clients map and set a value of true for it. We will fetch a greeting message to return to the client by calling the Greeting method on chatbot. Once we get the greeting (a string value), we call the SendMessage method passing in the client and the greeting converted to a byte slice. In the case that a pointer to a Client comes in over the hub's unregister channel, the hub will remove the entry in map for the given client and close the client's send channel, which signifies that the client won't be sending any more messages to the server. In the case that a pointer to a ClientMessage comes in over the hub's broadcastmsg channel, the hub will pass the client's message (as a string value) to the Reply method of the chatbot object. Once we get reply (a string value) from the agent, we call the SendMessage method passing in the client and the reply converted to a byte slice. The client type The Client type acts as a broker between Hub and the websocket connection. Here's what the Client struct looks like: type Client struct {  hub *Hub  conn *websocket.Conn send chan []byte } Each Client value contains a pointer to Hub, a pointer to a websocket connection, and a buffered channel, send, meant for outbound messages. The readPump method is responsible for relaying inbound messages coming in over the websocket connection to the hub: func (c *Client) readPump() { defer func() { c.hub.unregister <- c c.conn.Close() }() c.conn.SetReadLimit(maxMessageSize) c.conn.SetReadDeadline(time.Now().Add(pongWait)) c.conn.SetPongHandler(func(string) error { c.conn.SetReadDeadline(time.Now().Add(pongWait)); return nil }) for { _, message, err := c.conn.ReadMessage() if err != nil { if websocket.IsUnexpectedCloseError(err, websocket.CloseGoingAway) { log.Printf("error: %v", err) } break } message = bytes.TrimSpace(bytes.Replace(message, newline, space, -1)) // c.hub.broadcast <- message clientmsg := &ClientMessage{client: c, message: message} c.hub.broadcastmsg <- clientmsg } } We had to make a slight change to this function to fulfill the requirements of the live chat feature. In the Gorilla web chat example, the message alone was relayed over to Hub. Since we are directing chat bot responses, back to the client that sent them, not only do we need to send the message to the hub, but also the client that happened to send the message (shown in bold). We do so by creating a ClientMessage struct: type ClientMessage struct {  client *Client  message []byte } The ClientMessage struct contains fields to hold both the pointer to the client as well as the message, a byte slice. Going back to the readPump function in the client.go source file, the following two lines are instrumental in allowing the Hub to know which client sent the message: clientmsg := &ClientMessage{client: c, message: message}  c.hub.broadcastmsg <- clientmsg The writePump method is responsible for relaying outbound messages from the client's send channel over the websocket connection: func (c *Client) writePump() { ticker := time.NewTicker(pingPeriod) defer func() { ticker.Stop() c.conn.Close() }() for { select { case message, ok := <-c.send: c.conn.SetWriteDeadline(time.Now().Add(writeWait)) if !ok { // The hub closed the channel. c.conn.WriteMessage(websocket.CloseMessage, []byte{}) return } w, err := c.conn.NextWriter(websocket.TextMessage) if err != nil { return } w.Write(message) // Add queued chat messages to the current websocket message. n := len(c.send) for i := 0; i < n; i++ { w.Write(newline) w.Write(<-c.send) } if err := w.Close(); err != nil { return } case <-ticker.C: c.conn.SetWriteDeadline(time.Now().Add(writeWait)) if err := c.conn.WriteMessage(websocket.PingMessage, []byte{}); err != nil { return } } } } The ServeWS method is meant to be registered as an HTTP handler by the web application: func ServeWs(hub *Hub) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { conn, err := upgrader.Upgrade(w, r, nil) if err != nil { log.Println(err) return } client := &Client{hub: hub, conn: conn, send: make(chan []byte, 256)} client.hub.register <- client go client.writePump() client.readPump() }) } This method performs two important tasks. The method upgrades the normal HTTP connection to a websocket connection and registers the client to the hub. Now that we've set up the code for our web chat server, it's time to activate it in our web application. Activating the chat server In the igweb.go source file, we have included a function called startChatHub, which is responsible for starting the Hub: func startChatHub(hub *chat.Hub) {  go hub.Run() } We add the following code in the main function to create a new chatbot, associate it with the Hub, and start the Hub: chatbot := bot.NewAgentCase() hub := chat.NewHub(chatbot) startChatHub(hub) When we call the registerRoutes function to register all the routes for the server-side web application, note that we also pass in the hub value to the function: r := mux.NewRouter() registerRoutes(&env, r, hub) In the registerRoutes function, we need the hub to register the route handler for the Rest API endpoint that returns the agent's information: r.Handle("/restapi/get-agent-info", endpoints.GetAgentInfoEndpoint(env, hub.ChatBot())) The hub is also used to register the route handler for the WebSocket route, /ws. We register the ServeWS handler function, passing in the hub: r.Handle("/ws", chat.ServeWs(hub)) Now that we have everything in place to activate the chat server, it's time to focus on the star of the live chat feature—the chat agent. The agent's brain The chat bot type that we will use for the live chat feature, AgentCase, will implement the following Bot interface: type Bot interface { Greeting() string Reply(string) string Name() string Title() string ThumbnailPath() string SetName(string) SetTitle(string) SetThumbnailPath(string) } The Greeting method will be used to send an initial greeting to the user, enticing them to interact with the chat bot. The Reply method accepts a question (a string) and returns a reply (also a string) for the given question. The rest of the methods implemented are for purely psychological reasons to give humans the illusion that they are communicating with someone, rather than something. The Name method is a getter method that returns the chat bot's name. The Title method is a getter method that returns the chat bot's title. The ThumbnailPath method is a getter method that returns the path to the chat bot's avatar image. Each of the getter methods has a corresponding setter method: SetName, SetTitle, and SetThumbnailPath. By defining the Bot interface, we are clearly stating the expectations of a chat bot. This allows us to make the chat bot solution extensible in the future. For example, the intelligence that Case exhibits may be too rudimentary and limiting. In the near future, we may want to implement a bot named Molly, whose intelligence may be implemented using a more powerful algorithm. As long as the Molly chat bot implements the Bot interface, the new chat bot can be easily plugged into our web application. In fact, from the perspective of the server-side web application, it would just be a one-line code change. Instead of instantiating an AgentCase instance, we would instantiate an AgentMolly instance instead. Besides the difference in intelligence, the new chat bot, Molly, would come with its own name, title, and avatar image, so humans would be able to differentiate it from Case. Here's the AgentCase struct: type AgentCase struct { Bot name string title string thumbnailPath string knowledgeBase map[string]string knowledgeCorpus []string sampleQuestions []string } We have embedded the Bot interface to the struct definition, indicating that the AgentCase type will implement the Bot interface. The field name is for the name of the agent. The field title is for the title of the agent. The field thumbnailPath is used to specify the path to the chat bot's avatar image. The knowledgeBase field is  map of type map[string]string. This is essentially the agent's brain. Keys in the map are the common terms found in a particular question. Values in the map are the answers to the question. The knowledgeCorpus field, a string byte slice, is a knowledge corpus of the terms that may exist in questions that the bot will be asked. We use the keys of the knowledgeBase map to construct the knowledgeCorpus. A corpus is a collection of text that is used to conduct linguistic analysis. In our case, we will conduct the linguistic analysis based on the question (the query) that the human user provided to the bot. The sampleQuestions field, a string byte slice, will contain a list of sample questions that the user may ask the chat bot. The chat bot will provide the user with a sample question when it greets them to entice the human user into a conversation. It is understood that the human user is free to paraphrase the sample question or ask an entirely different question depending on their preference. The initializeIntelligence method is used to initialize Case's brain: func (a *AgentCase) initializeIntelligence() { a.knowledgeBase = map[string]string{ "isomorphic go isomorphic go web applications": "Isomorphic Go is the methodology to create isomorphic web applications using the Go (Golang) programming language. An isomorphic web application, is a web application, that contains code which can run, on both the web client and the web server.", "kick recompile code restart web server instance instant kickstart lightweight mechanism": "Kick is a lightweight mechanism to provide an instant kickstart to a Go web server instance, upon the modification of a Go source file within a particular project directory (including any subdirectories). An instant kickstart consists of a recompilation of the Go code and a restart of the web server instance. Kick comes with the ability to take both the go and gopherjs commands into consideration when performing the instant kickstart. This makes it a really handy tool for isomorphic golang projects.", "starter code starter kit": "The isogoapp, is a basic, barebones web app, intended to be used as a starting point for developing an Isomorphic Go application. Here's the link to the github page: https://github.com/isomorphicgo/isogoapp", "lack intelligence idiot stupid dumb dummy don't know anything": "Please don't question my intelligence, it's artificial after all!", "find talk topic presentation lecture subject": "Watch the Isomorphic Go talk by Kamesh Balasubramanian at GopherCon India: https://youtu.be/zrsuxZEoTcs", "benefits of the technology significance of the technology importance of the technology": "Here are some benefits of Isomorphic Go: Unlike JavaScript, Go provides type safety, allowing us to find and eliminate many bugs at compile time itself. Eliminates mental context-shifts between back- end and front-end coding. Page loading prompts are not necessary.", "perform routing web app register routes define routes": "You can implement client-side routing in your web application using the isokit Router preventing the dreaded full page reload.", "render templates perform template rendering": "Use template sets, a set of project templates that are persisted in memory and are available on both the server-side and the client-side", "cogs reusable components react-like react": "Cogs are reuseable components in an Isomorphic Go web application.", } a.knowledgeCorpus = make([]string, 1) for k, _ := range a.knowledgeBase { a.knowledgeCorpus = append(a.knowledgeCorpus, k) } a.sampleQuestions = []string{"What is isomorphic go?", "What are the benefits of this technology?", "Does isomorphic go offer anything react- like?", "How can I recompile code instantly?", "How can I perform routing in my web app?", "Where can I get starter code?", "Where can I find a talk on this topic?"} } There are three important tasks that occur within this method: First, we set Case's knowledge base. Second, we set Case's knowledge corpus. Third, we set the sample questions, which Case will utilize when greeting the human user. The first task we must take care of is to set Case's knowledge base. This consists of setting the knowledgeBase property of the AgentCase instance. As mentioned earlier, the keys in the map refer to terms found in the question, and the values in the map are the answers to the question. For example, the "isomorphic go isomorphic go web applications" key could service the following questions: What is Isomorphic Go? What can you tell me about Isomorphic Go? Due to the the large amount of text contained within the map literal declaration for the knowledgeBase map, I encourage you to view the source file, agentcase.go, on a computer. The second task we must take care of is to set Case's corpus, the collection of text used for linguistic analysis used against the user's question. The corpus is constructed from the keys of the knowledgeBase map. We set the knowledgeCorpus field property of the AgentCase instance to a newly created string byte slice using the built-in make function. Using a for loop, we iterate through all the entries in the knowledgeBase map and append each key to the knowledgeCorpus field slice. The third and last task we must take care of is to set the sample questions that Case will present to the human user. We simply populate the sampleQuestions property of the AgentCase instance. We use the string literal declaration to populate all the sample questions that are contained in the string byte slice. Here are the getter and setter methods of the AgentCase type: func (a *AgentCase) Name() string { return a.name } func (a *AgentCase) Title() string { return a.title } func (a *AgentCase) ThumbnailPath() string { return a.thumbnailPath } func (a *AgentCase) SetName(name string) { a.name = name } func (a *AgentCase) SetTitle(title string) { a.title = title } func (a *AgentCase) SetThumbnailPath(thumbnailPath string) { a.thumbnailPath = thumbnailPath } These methods are used to get and set the name, title, and thumbnailPath fields of the AgentCase object. Here's the constructor function used to create a new AgentCase instance: func NewAgentCase() *AgentCase {  agentCase := &AgentCase{name: "Case", title: "Resident Isomorphic  Gopher Agent",     thumbnailPath: "/static/images/chat/Case.png"}  agentCase.initializeIntelligence() return agentCase } We declare and initialize the agentCase variable with a new AgentCase instance, setting the fields for name, title, and thumbnailPath. We then call the initializeIntelligence method to initialize Case's brain. Finally, we return the newly created and initialized AgentCase instance. To summarize, we introduced you to the websocket package from the Gorilla toolkit project. We learned how to establish a persistent connection between the web server and the web client to create a server-side chatbot using WebSocket functionality. You read an excerpt from a book written by Kamesh Balasubramanian titled Isomorphic Go. In this book, you will learn how to build and deploy Isomorphic Go web applications. Top 4 chatbot development frameworks for developers How to create a conversational assistant or chatbot using Python Build a generative chatbot using recurrent neural networks (LSTM RNNs)    
Read more
  • 0
  • 0
  • 8435

article-image-app-and-web-development-in-2020-what-you-need-to-learn
Richard Gall
19 Dec 2019
6 min read
Save for later

App and web development in 2020: what you need to learn

Richard Gall
19 Dec 2019
6 min read
Web developers and app developers: not sure what you should be learning over the next 12 months? In such a fast-changing industry it can be hard to get a sense of which direction the proverbial wind is blowing, which means that when it comes to your future you can leave yourself in a somewhat weakened position.    So, if you’re looking for some ideas on how to organize your learning, look no further than this quick list.   Learn how to build microservices  Microservices are the dominant architectural model. There are many different reasons for this - the growth of APIs and web services, containerization - ultimately what’s important is that microservices allow you to build software in a way that’s modular. That promotes efficiency and agility which, in turn, empowers developers.   So, if you don’t yet have experience of developing microservices, you could do a lot worse than learning how to build them in 2020. Indeed, even if you don’t have a professional reason to learn more about microservices, exploring the topic through personal projects could be a real benefit to you in the future.   Find microservices content in Packt's range of cloud bundles. Learn a new language   This one might not be at the top of your agenda. If you’ve been working with JavaScript or Java for years, it seems strange to make time to learn something completely new. Surely, you’re probably thinking, I’d rather invest my time and energy into learning something that I can use at work?    In fact, learning a new language might just be one of the best things you can do in 2020. Not only will it give your resume a gentle kick, and potentially open up new opportunities for you in the future, it will also give you a better holistic understanding of how programming languages work, what their relative limitations and advantages over one another are. It’s a cliche that travel broadens the mind but when it comes to exploring and adventuring into new languages it’s certainly true.    But what should you learn? That, really is down to you, what you’re background is, and where you want to go. Some options are obvious - if you’re a Java developer Kotlin is the natural next step. Sometimes it’s not so obvious - JavaScript developers, for example might want to learn Go if they’re becoming more familiar with backend development, maybe even Rust or C++ if they’re feeling particularly adventurous and ambitious.  Explore new programming languages. Click here to find Packt's programming bundles of eBooks and videos. Learn a new framework Even if you don’t think learning a new language is appropriate for you, learning a new framework might be a more practical option that you can begin to use immediately. In the middle of the decade, when Angular.js was gaining traction across web development there was a lot of debate and discussion about the ‘best’ framework. Fortunately, this discourse has declined as it’s become clearer that choosing a framework is really about the best tool for the job, not a badge of personal identity. With this change, it means being open to learning new tools and frameworks will help you not only in terms of building out your resume, but also in terms of having a wide range of options for solving problems in your day to day work. Packt's web development bundles feature a range of eBooks on the most in-demand frameworks and tools. Explore them here. Re-learn a language you know Although it’s good to learn new things (obviously), we don’t talk enough about how valuable it can be to go back and learn something anew. This is for a couple of different reasons: on the one hand, it’s good to be able to review your level of understanding and to get up to speed with new features and capabilities that you might not have previously known about, but it also gives you a chance to explore a language from scratch and try to uncover a new perspective on it. This is particularly useful if you want to learn a new paradigm, like functional programming. Going back to core principles and theory is essential if you’re to unlock new levels of performance and control. Although it’s easy to dismiss theory as something academic, let’s make 2020 the year we properly begin to appreciate the close kinship between theory and practice. Learn how to go about learning It’s a given that being a developer means lifelong learning. And while we encourage you to take our advice and be open to new frameworks, microservices, and new programming languages, ultimately what you learn starts and ends with you. This isn’t easy - in fact, it’s probably one of the hardest aspects of working in technology full stop. That’s why, in 2020, make it your goal to learn more about learning. This article written by Jenn Schiffer is a couple of years old now but it articulates this challenge very well. She focuses mainly on some of the anxieties around learning new things and entering new communities, but the overarching point - that the conversation around how and why technologies should be used needs to be clearer - is a good one. True, there’s not much you can do about poor documentation or a toxic community, but you can think about your learning in a way that’s both open minded and well structured. For example, think about what you want or need to do - be reflective about your work and career. Be inquisitive and exploratory in your research - talk to people you know and trust, and read opinions and experiences from people you’ve never even heard of. By adopting a more intentional approach to the things you learn and the way you learn about them you’ll find that you’ll not only be able to learn new skills more efficiently, you’ll also enjoy it more too. Explore Packt’s full range of eBooks and videos on Packt store.
Read more
  • 0
  • 0
  • 8430

article-image-depth-look-ansible-plugins
Packt
15 Feb 2016
9 min read
Save for later

An In-depth Look at Ansible Plugins

Packt
15 Feb 2016
9 min read
In this article by Rishabh Das, author of the book Extending Ansible, we will deep dive into what Ansible plugins are and how you can write your own custom Ansible plugin. The article will discuss the different types of Ansible plugins in detail and explore them on a code level. The article will walk you through the Ansible Python API and using the extension points to write your own Ansible plugins. (For more resources related to this topic, see here.) Lookup plugins Lookup plugins are designed to read data from different sources and feed them to Ansible. The data source can be either from the local filesystem on the controller node or from an external data source. These may also be for file formats that are not natively supported by Ansible. If you decide to write your own lookup plugin, you need to drop it in one of the following directories for Ansible to pick it up during the execution of an Ansible playbook. A directory named lookup_plugins in the project root At ~/.ansible/plugins/lookup_plugins/ At /usr/share/ansible_plugins/lookup_plugins/ By default, a number of lookup plugins are already available in Ansible. Let's discuss some of the commonly used lookup plugins. Lookup plugin – file This is the most basic type of lookup plugin available in Ansible. It reads through the file content on the controller node. The data read from the file can then be fed to the Ansible playbook as a variable. In the most basic form, usage of file lookup is demonstrated in the following Ansible playbook: --- - hosts: all vars: data: "{{ lookup('file', './test-file.txt') }}" tasks: - debug: msg="File contents {{ data }}" The preceding playbook will read data off a local file, test-file.txt, from the playbook's root directory into a data variable. This variable is then fed to the task : debug module, which uses the data variable to print it onscreen. Lookup plugin – csvfile The csvfile lookup plugin was designed to read data from a CSV file on the controller node. This lookup module is designed to take in several parameters, which are discussed in this table: Parameter Default value Description file ansible.csv This is the file to read data from delimiter TAB This is the delimiter used in the CSV file, usually ','. col 1 This is the column number (index) default Empty string This returns this value if the requested key is not found in the CSV file Let's take an example of reading data from the following CSV file. The CSV file contains population and area details of different cities: File: city-data.csv City, Area, Population Pune, 700, 2.5 Million Bangalore, 741, 4.3 Million Mumbai, 603, 12 Million This file lies in the controller node at the root of the Ansible play. To read off data from this file, the csvfile lookup plugin is used. The following Ansible play tries to read the population of Mumbai from the preceding CSV file: Ansible Play – test-csv.yaml --- - hosts: all tasks: - debug: msg="Population of Mumbai is {{lookup('csvfile', 'Mumbai file=city-data.csv delimiter=, col=2')}}"   Lookup plugin – dig The dig lookup plugin can be used to run DNS queries against Fully Qualified Domain Name (FQDN). You can customize the lookup plugin's output using the different flags that are supported by the plugin. In the most basic form, it returns the IP of the given FQDN. This plugin has a dependency on the python-dns package. This should be installed on the controller node. The following Ansible play explains how to fetch the TXT record for any FQDN: --- - hosts: all tasks: - debug: msg="TXT record {{ lookup('dig', 'yahoo.com./TXT') }}" - debug: msg="IP of yahoo.com {{lookup('dig', 'yahoo.com', wantlist=True)}}" The preceding Ansible play will fetch the TXT records in step one and the IPs associated with FQDN yahoo.com in second. It is also possible to perform reverse DNS lookups using the dig plugin with the following syntax: - debug: msg="Reverse DNS for 8.8.8.8 is {{ lookup('dig', '8.8.8.8/PTR') }}" Lookup plugin – ini The ini lookup plugin is designed to read data off an .ini file. The INI file, in general, is a collection of key-value pairs under defined sections. The ini lookup plugin supports the following parameters: Parameter Default value Description type ini This is the type of file. It currently supports two formats: ini and property. file ansible.ini This is the name of file to read data from. section global This is the section of the ini file from which the specified key needs to be read. re false If the key is a regular expression, we need to set this to true. default Empty string If the requested key is not found in the ini file, we need to return this.   Taking an example of the following ini file, let's try to read some keys using the ini lookup plugin. The file is named network.ini: [default] bind_host = 0.0.0.0 bind_port = 9696 log_dir = /var/log/network [plugins] core_plugin = rdas-net firewall = yes The following Ansible play will read off the keys from the ini file: --- - hosts: all tasks: - debug: msg="core plugin {{ lookup('ini', 'core_plugin file=network.ini section=plugins') }}" - debug: msg="core plugin {{ lookup('ini', 'bind_port file=network.ini section=default') }}" The ini lookup plugin can also be used to read off values through a file that does not contain sections—for instance, a Java property file. Loops – lookup plugins for Iteration There are times when you need to perform the same task over and over again. It might be the case of installing various dependencies for a package or multiple inputs that go through the same operation—for instance, while checking and starting various services. Just like any other programming language provides a way to iterate over data to perform repetitive tasks, Ansible also provides a clean way to carry out the same operation. The concept is called looping and is provided by Ansible lookup plugins. Loops in Ansible are generally identified as those starting with “with_”. Ansible supports a number of looping options. A few of the most commonly used are discussed in the following section. Standard loop – with_items This is the simplest and most commonly used loop in Ansible. It is used to iterate over an item list and perform some operation on it. The following Ansible play demonstrates the use of the with_items lookup loop: --- - hosts: all tasks: - name: Install packages yum: name={{ item }} state=present with_items: - vim - wget - ipython The with_items lookup loop supports the use of hashes in which you can access the variables using the .<keyname> item in the Ansible playbook. The following playbook demonstrates the use of with_item to iterate over a given hash: --- - hosts: all tasks: - name: Create directories with specific permissions file: path={{item.dir}} state=directory mode={{item.mode | int}} with_items: - { dir: '/tmp/ansible', mode: 755 } - { dir: '/tmp/rdas', mode: 755 } The preceding playbook will create two directories with the specified permission sets. If you look closely while accessing the mode key from item, there exists a | int filter. This is a jinja2 filter, which is used to convert a string to an integer. DoUntill loop – until This has the same implementation as that in any other programming language. It executes at least once and keeps executing unless a specific condition is reached, as follows: < Code to follow > Creating your own lookup plugin This article covered some already available Ansible lookup plugins and explained how these can be used. This section will try to replicate a functionality of the dig lookup to get the IP address of a given FQDN. This will be done without using the dnspython library and will use the basic socket library for Python. The following example is only a demonstration of how you can write your own Ansible lookup plugin: import socket class LookupModule(object): def __init__(self, basedir=None, **kwargs): self.basedir = basedir def run(self, hostname, inject=None, **kwargs): hostname = str(hostname) try: host_detail = socket.gethostbyname(hostname) except: host_detail = 'Invalid Hostname' return host_detail The preceding code is a lookup plugin; let’s call it hostip. As you can note, there exists a class named LookupModule. Ansible identifies a Python file or module as a lookup plugin only when there exists a class called LookupModule. The module takes an argument hostname and checks whether there exists an IP corresponding to it—that is, whether it can be resolved to a valid IP address. If yes, it returns the IP address of the requested FQDN. If not, it returns Invalid Hostname. To use this module, place it in the lookup_modules directory at the root of the Ansible play. The following playbook demonstrates how you can use the hostip lookup just created: --- - hosts: all tasks: - debug: msg="{{lookup('hostip', item, wantlist=True)}}" with_items: - www.google.co.in - saliux.wordpress.com - www.twitter.com The preceding play will loop through the list of websites and pass it as an argument to the hostip lookup plugin. This will in turn return the IP associated with the requested domain. If you notice, there is an argument wantlist=True also passed while the hostip lookup plugin was called. This is to handle multiple outputs; that is, if there are multiple values associated with the requested domain, the values will be returned as a list. This makes it easy to iterate over the output values. Summary This article picked up on how the Ansible Python API for plugins is implemented in various Ansible plugins. The article discussed various types of plugins in detail, both from the implementation point of view and on a code level. The article also demonstrated how to write sample plugins by writing custom lookup plugins. By the end of this article, you should be able to write your own custom plugin for Ansible. Resources for Article: Further resources on this subject: Mastering Ansible – Protecting Your Secrets with Ansible [article] Ansible – An Introduction [article] Getting Started with Ansible [article]
Read more
  • 0
  • 0
  • 8428

article-image-interacting-gnu-octave-operators
Packt
20 Jun 2011
6 min read
Save for later

Interacting with GNU Octave: Operators

Packt
20 Jun 2011
6 min read
GNU Octave Beginner's Guide Become a proficient Octave user by learning this high-level scientific numerical tool from the ground up The reader will benefit from the previous article on GNU Octave Variables. Basic arithmetic Octave offers easy ways to perform different arithmetic operations. This ranges from simple addition and multiplication to very complicated linear algebra. In this section, we will go through the most basic arithmetic operations, such as addition, subtraction, multiplication, and left and right division. In general, we should think of these operations in the framework of linear algebra and not in terms of arithmetic of simple scalars. Addition and subtraction We begin with addition. Time for action – doing addition and subtraction operations I have lost track of the variables! Let us start afresh and clear all variables first: octave:66> clear (Check with whos to see if we cleared everything). Now, we define four variables in a single command line(!) octave:67> a = 2; b=[1 2 3]; c=[1; 2; 3]; A=[1 2 3; 4 5 6]; Note that there is an important difference between the variables b and c; namely, b is a row vector, whereas c is a column vector. Let us jump into it and try to add the different variables. This is done using the + character: octave:68>a+a ans = 4 octave:69>a+b ans = 3 4 5 octave:70>b+b ans = 2 4 6 octave:71>b+c error: operator +: nonconformant arguments (op1 is 1x3, op2 is 3x1) It is often convenient to enter multiple commands on the same line. Try to test the difference in separating the commands with commas and semicolons. What just happened? The output from Command 68 should be clear; we add the scalar a with itself. In Command 69, we see that the + operator simply adds the scalar a to each element in the b row vector. This is named element-wise addition. It also works if we add a scalar to a matrix or a higher dimensional array. Now, if + is applied between two vectors, it will add the elements together element-wise if and only if the two vectors have the same size, that is, they have same number of rows or columns. This is also what we would expect from basic linear algebra. From Command 70 and 71, we see that b+b is valid, but b+c is not, because b is a row vector and c is a column vector—they do not have the same size. In the last case, Octave produces an error message stating the problem. This would also be a problem if we tried to add, say b with A: octave:72>b+A error: operator +: nonconformant arguments (op1 is 1x3, op2 is 2x3) From the above examples, we see that adding a scalar to a vector or a matrix is a special case. It is allowed even though the dimensions do not match! When adding and subtracting vectors and matrices, the sizes must be the same. Not surprisingly, subtraction is done using the - operator. The same rules apply here, for example: octave:73> b-b ans = 0 0 0 is fine, but: octave:74> b-c error: operator -: nonconformant arguments (op1 is 1x3, op2 is 2x3) produces an error. Matrix multiplication The * operator is used for matrix multiplication. Recall from linear algebra that we cannot multiply any two matrices. Furthermore, matrix multiplication is not commutative. For example, consider the two matrices: The matrix product AB is defined, but BA is not. If A is size n x k and B has size k x m, the matrix product AB will be a matrix with size n x m. From this, we know that the number of columns of the "left" matrix must match the number of rows of the "right" matrix. We may think of this as (n x k)(k x m) = n x m. In the example above, the matrix product AB therefore results in a 2 x 3 matrix: Time for action – doing multiplication operations Let us try to perform some of the same operations for multiplication as we did for addition: octave:75> a*a ans = 4 octave:76> a*b ans = 2 4 6 octave:77> b*b error: operator *: nonconformant arguments (op1 is 1x3, op2 is 1x3) octave:78> b*c ans = 14 What just happened? From Command 75, we see that * multiplies two scalar variables just like standard multiplication. In agreement with linear algebra, we can also multiply a scalar by each element in a vector as shown by the output from Command 76. Command 77 produces an error—recall that b is a row vector which Octave also interprets as a 1 x 3 matrix, so we try to perform the matrix multiplication (1 x 3)(1 x 3), which is not valid. In Command 78, on the other hand, we have (1 x 3)(3 x 1) since c is a column vector yielding a matrix with size 1 x 1, that is, a scalar. This is, of course, just the dot product between b and c. Let us try an additional example and perform the matrix multiplication between A and B discussed above. First, we need to instantiate the two matrices, and then we multiply them: octave:79> A=[1 2; 3 4]; B=[1 2 3; 4 5 6]; octave:80> A*B ans = 9 12 15 19 26 33 octave:81> B*A error: operator *: nonconformant arguments (op1 is 2x3, op2 is 2x2) Seems like Octave knows linear algebra! Element-by-element, power, and transpose operations If the sizes of two arrays are the same, Octave provides a convenient way to multiply the elements element-wise. For example, for B: octave:82> B.*B ans = 1 4 9 16 25 36 Notice that the period (full stop) character precedes the multiplication operator. The period character can also be used in connection with other operators. For example: octave:83> B.+B ans = 2 4 6 8 10 12 which is the same as the command B+B. If we wish to raise each element in B to the power 2.1, we use the element-wise power operator.ˆ: octave:84> B.^2.1 ans = 1.0000 4.2871 10.0451 18.3792 29.3655 43.0643 You can perform element-wise power operation on two matrices as well (if they are of the same size, of course): octave:85> B.^B ans = 1 4 27 256 3125 46656 If the power is a real number, you can use ˆ instead of .ˆ; that is, instead of Command 84 above, you can use: octave:84>Bˆ2.1 Transposing a vector or matrix is done via the 'operator. To transpose B, we simply type: octave:86> B' ans = 1 4 2 5 3 6 Strictly, the ' operator is a complex conjugate transpose operator. We can see this in the following examples: octave:87> B = [1 2; 3 4] + I.*eye(2) B = 1 + 1i 2 + 0i 3 + 0i 4 + 1i octave:88> B' ans = 1 - 1i 3 - 0i 2 - 0i 4 - 1i Note that in Command 87, we have used the .* operator to multiply the imaginary unit with all the elements in the diagonal matrix produced by eye(2). Finally, note that the command transpose(B)or the operator .' will transpose the matrix, but not complex conjugate the elements.
Read more
  • 0
  • 0
  • 8422

article-image-creating-customized-dialog-boxes-wix
Packt
22 Oct 2010
5 min read
Save for later

Creating Customized Dialog Boxes with WiX

Packt
22 Oct 2010
5 min read
        Read more about this book       The WiX toolset ships with several User Interface wizards that are ready to use out of the box. We'll briefly discuss each of the available sets and then move on to learning how to create your own from scratch. In this article by Nick Ramirez, author of the book WiX: A Developer's Guide to Windows Installer XML, you'll learn about: Adding dialogs into the InstallUISequence Linking one dialog to another to form a complete wizard Getting basic text and window styling working Including necessary dialogs like those needed to display errors (For more resources on WiX, see here.) WiX standard dialog sets The wizards that come prebuilt with WiX won't fit every need, but they're a good place to get your feet wet. To add any one of them, you first have to add a project reference to WixUIExtension.dll, which can be found in the bin directory of your WiX program files. Adding this reference is sort of like adding a new source file. This one contains dialogs. To use one, you'll need to use a UIRef element to pull the dialog into the scope of your project. For example, this line, anywhere inside the Product element, will add the "Minimal" wizard to your installer: <UIRef Id="WixUI_Minimal" /> It's definitely minimal, containing just one screen. It gives you a license agreement, which you can change by adding a WixVariable element with an Id of WixUILicenseRtf and a Value attribute that points to a Rich Text Format (.rtf) file containing your new license agreement: <WixVariable Id="WixUILicenseRtf" Value="newLicense.rtf" /> You can also override the background image (red wheel on the left, white box on the right) by setting another WixVariable called WixUIDialogBmp to a new image. The dimensions used are 493x312. The other available wizards offer more and we'll cover them in the following sections. WixUI_Advanced The "Advanced" dialog set offers more: It has a screen that lets the user choose to install for just the current user or for all users, another where the end user can change the folder that files are installed to and a screen with a feature tree where features can be turned on or off. As in the following screenshot: You'll need to change your UIRef element to use WixUI_Advanced. This can be done by adding the following line: <UIRef Id="WixUI_Advanced" /> You'll also have to make sure that your install directory has an Id of APPLICATIONFOLDER, as in this example: <Directory Id="TARGETDIR" Name="SourceDir"> <Directory Id="ProgramFilesFolder"> <Directory Id="APPLICATIONFOLDER" Name="My Program" /> </Directory></Directory> Next, set two properties: ApplicationFolderName and WixAppFolder. The first sets the name of the install directory as it will be displayed in the UI. The second sets whether this install should default to being per user or per machine. It can be either WixPerMachineFolder or WixPerUserFolder. <Property Id="ApplicationFolderName" Value="My Program" /><Property Id="WixAppFolder" Value="WixPerMachineFolder" /> This dialog uses a bitmap that the Minimal installer doesn't: the white banner at the top. You can replace it with your own image by setting the WixUIBannerBmp variable. Its dimensions are 493x58. It would look something like this: <WixVariable Id="WixUIBannerBmp" Value="myBanner.bmp" /> WixUI_FeatureTree The WixUI_FeatureTree wizard shows a feature tree like the Advanced wizard, but it doesn't have a dialog that lets the user change the install path. To use it, you only need to set the UIRef to WixUI_FeatureTree, like so: <UIRef Id="WixUI_FeatureTree" /> This would produce a window that would allow you to choose features as show in the following screenshot: Notice that in the image, the Browse button is disabled. If any of your Feature elements have the ConfigurableDirectory attribute set to the Id of a Directory element, then this button will allow you to change where that feature is installed to. The Directory element's Id must be all uppercase. WixUI_InstallDir WixUI_InstallDir shows a dialog where the user can change the installation path. Change the UIRef to WixUI_InstallDir. Like so: <UIRef Id="WixUI_InstallDir" /> Here, the user can chose the installation path. This is seen in the following screenshot: You'll have to set a property called WIXUI_INSTALLDIR to the Id you gave your install directory. So, if your directory structure used INSTALLLDIR for the Id of the main install folder, use that as the value of the property. <Directory Id="TARGETDIR" Name="SourceDir"> <Directory Id="ProgramFilesFolder"> <Directory Id="INSTALLDIR" Name="My Program" /> </Directory></Directory> <Property Id="WIXUI_INSTALLDIR" Value="INSTALLDIR" /> WixUI_Mondo The WixUI_Mondo wizard gives the user the option of installing a "Typical", "Complete" or "Custom" install. Typical sets the INSTALLLEVEL property to 3 while Complete sets it to 1000. You can set the Level attribute of your Feature elements accordingly to include them in one group or the other. Selecting a Custom install will display a feature tree dialog where the user can choose exactly what they want. To use this wizard, change your UIRef element to WixUI_Mondo. <UIRef Id="WixUI_Mondo" /> This would result in a window like the following:
Read more
  • 0
  • 0
  • 8419
Unlock access to the largest independent learning library in Tech for FREE!
Get unlimited access to 7500+ expert-authored eBooks and video courses covering every tech area you can think of.
Renews at £15.99/month. Cancel anytime
article-image-rendering-cube-map-threejs
Soham Kamani
02 May 2016
7 min read
Save for later

Rendering a Cube Map in Three.js

Soham Kamani
02 May 2016
7 min read
Three.js is an awesome library. It makes complicated things such as 3D graphics and shaders easy for the average frontend developer, which opens up a lot of previously inaccessible avenues for web development. You can check out this repository of examples to see what's possible (basically, the sky is the limit). Even though Three.js provides all of this funcitonality, it is one of the lesser-documented libraries, and hence can be a little bit overwhelming for a newcomer. This tutorial will take you through the steps required to render a nice little scene with the Three.js library. Project Structure and Prerequisites We will be using npm and webpack to make our application, along with ES6 syntax. Initialize a new project in a new folder: npm init After that, install Three.js: npm install --save three And we are all set! Making a Cubemap A cubemap, if you haven't heard of it before, is precisely what its name suggests. Think of six enormous square pictures, all joined together to form a cube, with you being inside of the cube. The six pictures then form a cubemap. It is used to make 3D background sceneries and fillers. Every rendering in 3D graphics has two elements: the scene and the camera. The renderer then renders the scene relative to the camera. In this way, you can move through a scene by adjusting its camera, and at the same time stay still with respect to another scene because of its camera. This is the basic principle used while making movement-based 3D graphics. You (or in this case, your camera) are standing still with respect to the background (or in some cases, moving really slowly), considering it to be at a near-infinite distance from you. At the same time, you will be moving with respect to the objects around you, since they are considered to be within your immediate distance. First, we have to import our dependencies and initialize important objects. As mentioned before, each rendering will have a scene and a camera. Three.js provides constructors for both of these things. The arguments we see for the camera are parameters such as frustum vertical, aspect ratio, near frame, and far frame, which are much beyond the scope of this post. For almost all cases, these numbers can be consdered as defaults and need not be changed. import THREE from 'three'; let sceneCube = new THREE.Scene(); let cameraCube = new THREE.PerspectiveCamera( 60, window.innerWidth / window.innerHeight, 1, 100000 ); Each cubemap is composed of six images. In the snippet below, we are just making an array of paths, which is where all our images are kept. In this case, the image for one side of our cube (the positive x side) would be located at the path '/cubemap/cm_x_p.jpg': let path = '/cubemap/cm'; let format = '.jpg'; let urls = [ path + '_x_p' + format, path + '_x_n' + format, path + '_y_p' + format, path + '_y_n' + format, path + '_z_p' + format, path + '_z_n' + format ]; But where do I find cubemap images from? Normally, you would have to use Google for cubemap images, but those are not the best quality. You can make your own cubemap from normal images and some respectable photoshop skills, or you can take images from some of the examples that already exist. This is the part where we actually "create" the cubemap from the images we have. Each scene is defined by a number of "meshes." Each mesh is defined by a geometry, which specifies the shape of the mesh, and a material, which specifies the appearence and coloring of the mesh. We now load the six images defined previously into a "texture cube," which is then used to define our material. The geometry we used is called a box geometry, and we are defining the length, width, and breadth of this box as 100 units each: let textureCube = THREE.ImageUtils.loadTextureCube(urls, THREE.CubeRefractionMapping); let shader = THREE.ShaderLib.cube; shader.uniforms.tCube.value = textureCube; let material = new THREE.ShaderMaterial({ fragmentShader: shader.fragmentShader, vertexShader: shader.vertexShader, uniforms: shader.uniforms, depthWrite: false, side: THREE.BackSide }), mesh = new THREE.Mesh(new THREE.BoxGeometry(100, 100, 100), material); Finally, our cubemap is added to the scene: sceneCube.add(mesh); Of course, we want to keep our code modular, so all of the above code for making a cubemap should ideally be wrapped in its own function and used in our main program as and when it is needed. The final cubemap "module" would look something like this: 'use strict'; import THREE from 'three'; let Cubemap = function () { let sceneCube = new THREE.Scene(); let cameraCube = new THREE.PerspectiveCamera( 60, window.innerWidth / window.innerHeight, 1, 100000 ); let path = '/cubemap/cm'; let format = '.jpg'; let urls = [ path + '_x_p' + format, path + '_x_n' + format, path + '_y_p' + format, path + '_y_n' + format, path + '_z_p' + format, path + '_z_n' + format ]; let textureCube = THREE.ImageUtils.loadTextureCube(urls, THREE.CubeRefractionMapping); let shader = THREE.ShaderLib.cube; shader.uniforms.tCube.value = textureCube; let material = new THREE.ShaderMaterial({ fragmentShader: shader.fragmentShader, vertexShader: shader.vertexShader, uniforms: shader.uniforms, depthWrite: false, side: THREE.BackSide }), mesh = new THREE.Mesh(new THREE.BoxGeometry(100, 100, 100), material); sceneCube.add(mesh); return { scene : sceneCube, camera : cameraCube }; }; module.exports = Cubemap; Making Our Core Module Now, we will have to write the core of our little app to actually render the cubemap onto an element in the web browser: We import the dependencies to be used and initialize our WebGL renderer:   'use strict'; import THREE from 'three'; import Cubemap from './Cubemap'; let scene = new THREE.Scene(); let camera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 10000); let renderer = new THREE.WebGLRenderer(); renderer.autoClear = false; renderer.setSize(window.innerWidth, window.innerHeight); document.body.appendChild(renderer.domElement); The renderer returns a canvas element, which you can fix to your DOM. This canvas element is where all the magic happens. As usual, we are creating another scene and camera, different from the ones in our cubemap. All objects that are not the cubemap will be included in this scene. Next, we add ambient light to our scene. This is so that we can actually see the other objects we add in the scene: let lightAmbient = new THREE.AmbientLight(0x202020); // soft white light scene.add(lightAmbient); We define and start our rendering function: let cubemap = Cubemap(); let render = function () { requestAnimationFrame(render); renderer.render(cubemap.scene, cubemap.camera); renderer.render(scene, camera); cubemap.camera.rotation.copy(camera.rotation); }; render(); In case you are using movement (which is most often the case with WebGL), you will want to render your scene a number of times a second. The requestAnimationFrame function is a native browser function that calls the function you pass to it after a set time. To really see your cubemapscene come alive, add a moving element as shown in the Hello World example from the Three.js website. Three.js may seem overwhelming at first, but it's a huge improvement over the otherwise steep learning curve for GLSL. If you want to see a slightly more complex example of using cubemaps and objects in Three.js, you can go here. About the author Soham Kamai is a Fullstack web developer and electronics hobbyist. He is especially interested in JavaScript, Python, and IOT. He can be found on Twitter @sohamkamani and GitHub at https://github.com/sohamkamani.
Read more
  • 0
  • 0
  • 8419

article-image-how-to-build-a-relay-react-app-tutorial
Bhagyashree R
01 Jan 2019
12 min read
Save for later

How to build a Relay React App [Tutorial]

Bhagyashree R
01 Jan 2019
12 min read
Relay is used with both web and mobile React applications. It relies on a language called GraphQL which is used to fetch resources and to mutate those resources. The premise of Relay is that it can be scaled in ways that Redux and other approaches to handling state are limiting. It does this by eliminating them and keeping the focus on the data requirements of the component. In this article, we will build a Todo React Native application using Relay. By the end of this article, you should feel comfortable about how data moves around in a GraphQL centric architecture. At a high level, you can think of Relay as an implementation of Flux architecture patterns, and you can think of GraphQL as the interface that describes how the Flux stores within Relay work. At a more practical level, the value of Relay is ease of implementation. For example, with Redux, you have a lot of implementation work to do, just to populate the stores with data. This gets verbose over time. It's this verbosity that makes Redux difficult to scale beyond a certain point. [box type="shadow" align="" class="" width=""]This article is taken from the book React and React Native - Second Edition by Adam Boduch. This book guides you through building applications for web and native mobile platforms with React, JSX, Redux, and GraphQL.  To follow along with the examples implemented in this article, you can find the code in the GitHub repository of the book. [/box] TodoMVC and Relay TodoMVC example for Relay will be a robust yet concise example. We will walk through an example React Native implementation of a Todo app. The key is that it'll use the same GraphQL backend as the web UI. I've included the web version of the TodoMVC app in the code that ships with this book, but I won't dwell on the details of how it works. If you've worked on web development in the past 5 years, you've probably come across a sample Todo app. Here's what the web version looks like: Even if you haven't used any of the TodoMVC apps before, I would recommend playing with this one before trying to implement the native version, which is what you'll be doing for the remainder of the article. The goal of the native version that you're about to implement isn't functional parity. In fact, you're shooting for a very minimal subset of todo functionality. The aim is to show you that Relay works mostly the same on native platforms as it does on web platforms and that the GraphQL backend can be shared between web and native apps. The GraphQL schema The schema is the vocabulary used by GraphQL backend server and the Relay components in the frontend. The GraphQL type system enables the schema to describe the data that's available, and how to put it all together when a query request comes in. This is what makes the whole approach so scalable, the fact that the GraphQL runtime figures out how to put data together. All you need to supply are functions that tell GraphQL where the data is; for example, in a database or in some remote service endpoint. Let's take a look at the types used in the GraphQL schema for the TodoMVC app: You can find the code in this section on GitHub. import { GraphQLBoolean, GraphQLID, GraphQLInt, GraphQLList, GraphQLNonNull, GraphQLObjectType, GraphQLSchema, GraphQLString } from 'graphql'; import { connectionArgs, connectionDefinitions, connectionFromArray, cursorForObjectInConnection, fromGlobalId, globalIdField, mutationWithClientMutationId, nodeDefinitions, toGlobalId } from 'graphql-relay'; import { Todo, User, addTodo, changeTodoStatus, getTodo, getTodos, getUser, getViewer, markAllTodos, removeCompletedTodos, removeTodo, renameTodo } from './database'; const { nodeInterface, nodeField } = nodeDefinitions( globalId => { const { type, id } = fromGlobalId(globalId); if (type === 'Todo') { return getTodo(id); } if (type === 'User') { return getUser(id); } return null; }, obj => { if (obj instanceof Todo) { return GraphQLTodo; } if (obj instanceof User) { return GraphQLUser; } return null; } ); const GraphQLTodo = new GraphQLObjectType({ name: 'Todo', fields: { id: globalIdField(), complete: { type: GraphQLBoolean }, text: { type: GraphQLString } }, interfaces: [nodeInterface] }); const { connectionType: TodosConnection, edgeType: GraphQLTodoEdge } = connectionDefinitions({ nodeType: GraphQLTodo }); const GraphQLUser = new GraphQLObjectType({ name: 'User', fields: { id: globalIdField(), todos: { type: TodosConnection, args: { status: { type: GraphQLString, defaultValue: 'any' }, ...connectionArgs }, resolve: (obj, { status, ...args }) => connectionFromArray(getTodos(status), args) }, numTodos: { type: GraphQLInt, resolve: () => getTodos().length }, numCompletedTodos: { type: GraphQLInt, resolve: () => getTodos('completed').length } }, interfaces: [nodeInterface] }); const GraphQLRoot = new GraphQLObjectType({ name: 'Root', fields: { viewer: { type: GraphQLUser, resolve: getViewer }, node: nodeField } }); const GraphQLAddTodoMutation = mutationWithClientMutationId({ name: 'AddTodo', inputFields: { text: { type: new GraphQLNonNull(GraphQLString) } }, outputFields: { viewer: { type: GraphQLUser, resolve: getViewer }, todoEdge: { type: GraphQLTodoEdge, resolve: ({ todoId }) => { const todo = getTodo(todoId); return { cursor: cursorForObjectInConnection(getTodos(), todo), node: todo }; } } }, mutateAndGetPayload: ({ text }) => { const todoId = addTodo(text); return { todoId }; } }); const GraphQLChangeTodoStatusMutation = mutationWithClientMutationId({ name: 'ChangeTodoStatus', inputFields: { id: { type: new GraphQLNonNull(GraphQLID) }, complete: { type: new GraphQLNonNull(GraphQLBoolean) } }, outputFields: { viewer: { type: GraphQLUser, resolve: getViewer }, todo: { type: GraphQLTodo, resolve: ({ todoId }) => getTodo(todoId) } }, mutateAndGetPayload: ({ id, complete }) => { const { id: todoId } = fromGlobalId(id); changeTodoStatus(todoId, complete); return { todoId }; } }); const GraphQLMarkAllTodosMutation = mutationWithClientMutationId({ name: 'MarkAllTodos', inputFields: { complete: { type: new GraphQLNonNull(GraphQLBoolean) } }, outputFields: { viewer: { type: GraphQLUser, resolve: getViewer }, changedTodos: { type: new GraphQLList(GraphQLTodo), resolve: ({ changedTodoIds }) => changedTodoIds.map(getTodo) } }, mutateAndGetPayload: ({ complete }) => { const changedTodoIds = markAllTodos(complete); return { changedTodoIds }; } }); const GraphQLRemoveCompletedTodosMutation = mutationWithClientMutationId( { name: 'RemoveCompletedTodos', outputFields: { viewer: { type: GraphQLUser, resolve: getViewer }, deletedIds: { type: new GraphQLList(GraphQLString), resolve: ({ deletedIds }) => deletedIds } }, mutateAndGetPayload: () => { const deletedTodoIds = removeCompletedTodos(); const deletedIds = deletedTodoIds.map( toGlobalId.bind(null, 'Todo') ); return { deletedIds }; } } ); const GraphQLRemoveTodoMutation = mutationWithClientMutationId({ name: 'RemoveTodo', inputFields: { id: { type: new GraphQLNonNull(GraphQLID) } }, outputFields: { viewer: { type: GraphQLUser, resolve: getViewer }, deletedId: { type: GraphQLID, resolve: ({ id }) => id } }, mutateAndGetPayload: ({ id }) => { const { id: todoId } = fromGlobalId(id); removeTodo(todoId); return { id }; } }); const GraphQLRenameTodoMutation = mutationWithClientMutationId({ name: 'RenameTodo', inputFields: { id: { type: new GraphQLNonNull(GraphQLID) }, text: { type: new GraphQLNonNull(GraphQLString) } }, outputFields: { todo: { type: GraphQLTodo, resolve: ({ todoId }) => getTodo(todoId) } }, mutateAndGetPayload: ({ id, text }) => { const { id: todoId } = fromGlobalId(id); renameTodo(todoId, text); return { todoId }; } }); const GraphQLMutation = new GraphQLObjectType({ name: 'Mutation', fields: { addTodo: GraphQLAddTodoMutation, changeTodoStatus: GraphQLChangeTodoStatusMutation, markAllTodos: GraphQLMarkAllTodosMutation, removeCompletedTodos: GraphQLRemoveCompletedTodosMutation, removeTodo: GraphQLRemoveTodoMutation, renameTodo: GraphQLRenameTodoMutation } }); export default new GraphQLSchema({ query: GraphQLRoot, mutation: GraphQLMutation }); There are a lot of things being imported here, so I'll start with the imports. I wanted to include all of these imports because I think they're contextually relevant for this discussion. First, there's the primitive GraphQL types from the graphql library. Next, you have helpers from the graphql-relay library that simplify defining a GraphQL schema. Lastly, there's imports from your own database module. This isn't necessarily a database, in fact, in this case, it's just mock data. You could replace database with api for instance, if you needed to talk to remote API endpoints, or we could combine the two; it's all GraphQL as far as your React components are concerned. Then, you define some of your own GraphQL types. For example, the GraphQLTodo type has two fields—text and complete. One is a Boolean and one is a string. The important thing to note about GraphQL fields is the resolve() function. This is how you tell the GraphQL runtime how to populate these fields when they're required. These two fields simply return property values. Then, there's the GraphQLUser type. This field represents the user's entire universe within the UI, hence the name. The todos field, for example, is how you query for todo items from Relay components. It's resolved using the connectionFromArray() function, which is a shortcut that removes the need for more verbose field definitions. Then, there's the GraphQLRoot type. This has a single viewer field that's used as the root of all queries. Now let's take a closer look at the add todo mutation, as follows. I'm not going to go over every mutation that's used by the web version of this app, in the interests of space: const GraphQLAddTodoMutation = mutationWithClientMutationId({ name: 'AddTodo', inputFields: { text: { type: new GraphQLNonNull(GraphQLString) } }, outputFields: { viewer: { type: GraphQLUser, resolve: getViewer }, todoEdge: { type: GraphQLTodoEdge, resolve: ({ todoId }) => { const todo = getTodo(todoId); return { cursor: cursorForObjectInConnection(getTodos(), todo), node: todo }; } } }, mutateAndGetPayload: ({ text }) => { const todoId = addTodo(text); return { todoId }; } }); All mutations have a mutateAndGetPayload() method, which is how the mutation actually makes a call to some external service to change the data. The returned payload can be the changed entity, but it can also include data that's changed as a side-effect. This is where the outputFields come into play. This is the information that's handed back to Relay in the browser so that it has enough information to properly update components based on the side effects of the mutation. Don't worry, you'll see what this looks like from Relay's perspective shortly. The mutation type that you've created here is used to hold all application mutations. Lastly, here's how the entire schema is put together and exported from the module: export default new GraphQLSchema({ query: GraphQLRoot, mutation: GraphQLMutation }); Don't worry about how this schema is fed into the GraphQL server for now. Bootstrapping Relay At this point, you have the GraphQL backend up and running. Now, you can focus on your React components in the frontend. In particular, you're going to look at Relay in a React Native context, which really only has minor differences. For example, in web apps, it's usually react-router that bootstraps Relay. In React Native, it's a little different. Let's look at the App.js file that serves as the entry point for your native app: You can find the code in this section on GitHub. Let's break down what's happening here, starting with the environment constant: const environment = new Environment({ network: Network.create({ schema }), store: new Store(new RecordSource()) }); This is how you communicate with the GraphQL backend, by configuring a network. In this example, you're importing Network from relay-local-schema, which means that no network requests are being made. This is really handy for when you're getting started—especially building a React Native app. Next, there's the QueryRenderer component. This Relay component is used to render other components that depend on GraphQL queries. It expects a query property: query={graphql` query App_Query($status: String!) { viewer { ...TodoList_viewer } } `} Note that queries are prefixed by the module that they're in. In this case, App. This query uses a GraphQL fragment from another module, TodoList, and is named TodoList_viewer. You can pass variables to the query: variables={{ status: 'any' }} Then, the render property is a function that renders your components when the GraphQL data is ready: If something went wrong, error will contain information about the error. If there's no error and no props, it's safe to assume that the GraphQL data is still loading. Adding todo items In the TodoInput component, there's a text input that allows the user to enter new todo items. When they're done entering the todo, Relay will need to send a mutation to the backend GraphQL server. Here's what the component code looks like: You can find the code in this section on GitHub. It doesn't look that different from your typical React Native component. The piece that stands out is the mutation—AddTodoMutation. This is how you tell the GraphQL backend that you want a new todo node created. Let's see what the application looks like so far: The textbox for adding new todo items is just above the list of todo items. Now, let's look at the TodoList component, which is responsible for rendering the todo item list. Rendering todo items It's the job of the TodoList component to render the todo list items. When AddTodoMutation takes place, the TodoList component needs to be able to render this new item. Relay takes care of updating the internal data stores where all of our GraphQL data lives. Here's a look at the item list again, with several more todos added: Here's the TodoList component itself: You can find the code in this section on GitHub. The relevant GraphQL to get the data you need is passed as a second argument to createFragmentContainer(). This is the declarative data dependency for the component. When you render the <Todo> component, you're passing it the edge.todo data. Now, let's see what the Todo component itself looks like. Completing todo items The last piece of this application is rendering each todo item and providing the ability to change the status of the todo. Let's take a look at this code: You can find the code in this section on GitHub. The actual component that's rendered is a switch control and the item text. When the user marks the todo as complete, the item text is styled as crossed off. The user can also uncheck items. The ChangeTodoStatusMutation mutation sends the request to the GraphQL backend to change the todo state. The GraphQL backend then talks to any microservices that are needed to make this happen. Then, it responds with the fields that this component depends on. The important part of this code that I want to point out is the fragments used in the Relay container. This container doesn't actually use them directly. Instead, they're used by the todos query in the TodoList component (Todo.getFrament()). This is useful because it means that you can use the Todo component in another context, with another query, and its data dependencies will always be satisfied. In this article, we implemented some specific Relay and GraphQL ideas. Starting with the GraphQL schema, we learned how to declare the data that's used by the application and how these data types resolve to specific data sources, such as microservice endpoints. Then, we learned about bootstrapping GraphQL queries from Relay in your React Native app. Next, we walked through the specifics of adding, changing and listing todo items. The application itself uses the same schema as the web version of the Todo application, which makes things much easier when you're developing web and native React applications. If you found this post useful, do check out the book, React and React Native - Second Edition. This book guides you through building applications for web and native mobile platforms with React, JSX, Redux, and GraphQL. JavaScript mobile frameworks comparison: React Native vs Ionic vs NativeScript React introduces Hooks, a JavaScript function to allow using React without classes npm JavaScript predictions for 2019: React, GraphQL, and TypeScript are three technologies to learn
Read more
  • 0
  • 0
  • 8416

article-image-analyzing-network-reconnaissance-attempts
Packt
11 Aug 2015
8 min read
Save for later

Analyzing network reconnaissance attempts

Packt
11 Aug 2015
8 min read
In this article by Piyush Verma, author of the book Wireshark Network Security, you will be introduced to using Wireshark to detect network reconnaissance activities performed by an insider. A dictionary definition of reconnaissance is "military observation of a region to locate an enemy or ascertain strategic features." A good analogy for reconnaissance will be a thief studying the neighborhood to observe which houses are empty and which are occupied, the number of family members who live at the occupied houses, their entrance points, the time during which these occupied houses are empty, and so on, before he/she can even think about stealing anything from that neighborhood. Network reconnaissance relates to the act of gathering information about the target’s network infrastructure, the devices that reside on the network, the platform used by such devices, and the ports opened on them, to ultimately come up with a brief network diagram of devices and then plan the attack accordingly. The tools required to perform network scanning are readily available and can be downloaded easily over the Internet. One such popular tool is Nmap, short for Network Mapper. It is written by Gordon “Fyodor” Lyon and is a popular tool of choice to perform network-based reconnaissance. Network scanning activities can be as follows: Scanning for live machines Port scans Detecting the presence of a firewall or additional IP protocols (For more resources related to this topic, see here.) Detect the scanning activity for live machines An attacker would want to map out the live machines on the network rather than performing any activity with an assumption that all the machines are live. The following are the two popular techniques, which can be used, and the ways to detect them using Wireshark. Ping sweep This technique makes use of a simple technique to ping an IP address in order to identify whether it is alive or not. Almost all modern networks block the ICMP protocol; hence, this technique is not very successful. However, in case your network supports ICMP-based traffic, you can detect this attack by looking for large number of ping requests going to a range of IP addresses on your network. A helpful filter in this case will be: icmp.type == 8 || icmp.type == 0 ICMP Type 8 = ECHO Request ICMP Type 0 = ECHO Reply ARP sweep ARP responses cannot be disabled on the network; hence, this technique works very well while trying to identify live machines on a local network. Using this technique, an attacker can discover hosts that may be hidden from other discovery methods, such as ping sweeps, by a firewall. To perform this, an attacker sends ARP broadcasts (destination MAC address: FF:FF:FF:FF:FF:FF) for all possible IP addresses on a given subnet, and the machines responding to these requests are noted as alive or active. To detect ARP sweep attempts, we need to look for a massive amount of ARP broadcasts from a client machine on the network. Another thing to note will be the duration in which these broadcasts are sent. These are highlighted in the following screenshot: Shows an ARP sweep in action A point to note is the source of these ARP requests to avoid false positives because such requests may also be made by legitimate services, such as SNMP. Identifying port scanning attempts Now, we will look at different port scanning techniques used by attackers and how to detect them using Wireshark. TCP Connect scan In a TCP Connect scan, a client/attacker sends a SYN packet to the server/victim on a range of port numbers. For the ports that respond to SYN/ACK, the client completes the 3-way handshake by sending an ACK and then terminates the connection by sending an RST to the server/victim, while the ports that are closed reply with RST/ACK packets to the SYN sent by the client/attacker. Hence, in order to identify this type of scan, we will need to look for a significantly large number of RST (Expert Info) or SYN/ACK packets. Generally, when a connection is established, some form of data is transferred; however, in scanning attempts, no data is sent across indicating a suspicious activity (Conversations | TCP). Another indication is the short period of time under which these packets are sent (Statistics | Flow Graph). Wireshark’s Flow Graph While observing the TCP flow in the Flow Graph, we noted a sequence of SYN, SYN/ACK, and ACKs along with SYN and RST/ACKs. Another indication is the fraction of seconds (displayed on left-hand side) under which these packets are sent. Shows a complete 3-way handshake with open-ports and how quickly the packets were sent under the ‘Time’ column Wireshark’s Expert Info Even the Expert Info window indicates a significant number of connection resets. Shows Warning Tab under Expert Info Wireshark’s Conversations We can look at the TCP Conversations, to observe which type of scan is underway and the number of Bytes associated with each conversation. Shows the number of packets and Bytes transferred for each Conversation The number 4 in the Packets column indicates a SYN, SYN/ACK, ACK, and RST packets, and the number 2 indicates the SYN sent by Nmap and RST/ACK received for a closed port. Stealth scan A stealth scan is different from the TCP Connect scan explained earlier and is never detected by the application layer as the complete TCP 3-way handshake is never established during this scan and hence also known as half-open scan. During this scan, a client/attacker sends a SYN packet to the server/victim on a range of port numbers. If Nmap receives a SYN/ACK to the SYN request, it means that the port is open, and Nmap then sends an RST to close the connection without ever completing the 3-way handshake, while the ports that are closed reply with RST/ACK packets to the SYN requests. The way to detect this attack is similar to the previous scan, where you will note a lot of RST (Expert Info) or SYN/ACK packets without data transfers (Conversations | TCP) on the network. Another indication is the short period of time under which these packets are sent (Statistics | Flow Graph). Now, we will look at the Flow Graph, Expert Info, and Conversations in Wireshark for Stealth scan. Wireshark’s Flow Graph While observing the TCP flow in the Flow Graph, we noted a sequence of SYN, SYN/ACK, and RSTs (indicating a half-open connection) along with SYN and RST/ACKs. Another indication is the fraction of seconds (displayed on the left-hand side) under which these packets are sent. Shows the half-open scan underway and how quickly the packets were sent under the "Time" column Wireshark’s Expert Info The huge number of connection resets is another indication of a scan underway. Shows Warning Tab under Expert Info Wireshark’s Conversations TCP Conversations also provide an insight to indicate that a half-open scan is underway and the number of Bytes associated with each attempt. Shows the number of packets and Bytes transferred for each Conversation The number 3 in the Packets column indicates a SYN, SYN/ACK, and RST packets, and the number 2 indicates the SYN sent by Nmap and RST/ACK received for a closed port. NULL scan During a null scan, unusual TCP packets are sent with no flags set. If the resultant of this is no response, it means that the port is either open or filtered; while the RST/ACK response means that the port is closed. A quick way to detect, whether such a scan is underway, is to filter on tcp.flags == 0x00. UDP scan The last three techniques were related to TCP-based scans. Many common protocols work over UDP as well (DNS, SNMP, TFTP, and so on), and scans are conducted to detect whether such ports are open or not. No response to a UDP port scan indicates that the port is either open or firewalled, and a response of ICMP Destination Unreachable/Port Unreachable means that the port is closed. Detect UDP scans by filtering on (icmp.type == 3) && (icmp.code == 3). ICMP Type 3 = Destination Unreachable ICMP Code 3 = Port Unreachable Other scanning attempts The following scanning techniques go beyond the traditional port scanning techniques and help the attacker in further enumeration of the network. ACK scan ACK Flag scan never locates an open port; rather, it only provides the result in form of "filtered" or "unfiltered" and is generally used to detect the presence of a firewall. No response means port is filtered, and RST response indicates that the port is unfiltered. Shows Flow Graph (TCP) of an ACK Flag Scan IP Protocol scan IP Protocol scan is conducted by attackers to determine the presence of additional IP protocols in use by the victim. For example, if a router is scanned using this technique, it might reveal the use of other protocols as EGP, IGP, EIGRP, and so on. No response indicates that a protocol is present or the response is filtered, while an ICMP Destination Unreachable/Protocol Unreachable indicates that the protocol is not supported by the device. To detect this scan using Wireshark, we can filter the traffic based on (icmp.type == 3) && (icmp.code == 2). ICMP Type 3 = Destination Unreachable ICMP Code 2 = Protocol Unreachable Summary In this article, we used Wireshark and the set of robust features it has to offer, to analyze the network scanning attempts performed by attackers. Resources for Article: Further resources on this subject: Wireshark [article] Wireshark: Working with Packet Streams [article] UDP/TCP Analysis [article]
Read more
  • 0
  • 0
  • 8413

article-image-how-to-work-with-the-intellij-idea-selenium-plugin
Amey Varangaonkar
03 Apr 2018
3 min read
Save for later

How to work with the Selenium IntelliJ IDEA plugin

Amey Varangaonkar
03 Apr 2018
3 min read
Most of the framework components you design and build will be customized to your application under test. However, there are many third-party tools and plugins available, which you can use to provide better results processing, reporting, performance, and services to engineers using the framework. In this article, we cover one of the most popular plugins used with Selenium - the Selenium IntelliJ IDEA plugin. IntelliJ IDEA Selenium plugin When we covered building page object classes earlier, we discussed how to define the locators on a page for each WebElement or MobileElement using the @findBy annotations. That required the user to use one of the Inspectors or plugins to view the DOM structure and hand-code a robust locator that is cross-platform safe. Now, when using CSS and XPath locators, the hierarchy of the element can get complex, and there is a greater chance of building invalid locators. So, Perfect Test has come up with a Selenium plugin for the IntelliJ IDEA that will find and create locators on the fly. Before discussing some of the features of the plugin, let's review where this is located. Sample project files There are instructions on the www.perfect-test.com site for installing the plugin and once that is done, users can create a new project using a sample template, which will auto- generate a series of template files. These files are generic "getting started" files, but you should still follow the structure and design of the framework as outlined in this book. Here is a quick screenshot of the autogenerated file structure of the sample project: Once the plugin is enabled by simply clicking on the Selenium icon in the toolbar, users can use the Code Generate menu features to create code samples, Java methods, getter/setter methods, WebElements, copyrights for files, locators, and so on. Generating element locators The plugin has a nice feature for creating WebElement definitions, adding locators of choice, and validating them in the class. It provides a set of tooltips to tell the user what is incorrect in the syntax of the locator, which is helpful when creating CSS and XPath strings. Here is a screenshot of the locator strategy feature: Once the WebElement structure is built into the page object class, you can capture and verify the locator, and it will indicate an error with a red underline. When moving over the invalid syntax, it provides a tooltip and a lightbulb icon to the left of it, where users can use features for Check Element Existence on page and Fix Locator Popup. These are very useful for quickly finding syntax errors and defining locators. Here is a screenshot of the Check Element Existence on page feature: Here is a screenshot of the Fix Locator Popup feature: The Selenium IntelliJ plugin deals mostly with creating locators and the differences between CSS and XPath syntax. The tool also provides drop-down lists of examples where users can pick and choose how to build the queries. It's a great way to get started using Selenium to build real page object classes, and it provides a tool to validate complex CSS and XPath structures in locators! Apart from the Selenium IntelliJ plugin, there are other third-party APIs such as HTML Publisher Plugin, BrowserMob Proxy Plugin, ExtentReports Reporter API and also Sauce Labs Test Cloud services.  This article is an excerpt taken from the book Selenium Framework Design in Data-Driven Testing by Carl Cocchiaro. It presents a step-by-step approach to design and build a data-driven test framework using Selenium WebDriver, Java, and TestNG.  
Read more
  • 0
  • 0
  • 8407
article-image-lookups
Packt
17 Dec 2014
24 min read
Save for later

Mastering Splunk: Lookups

Packt
17 Dec 2014
24 min read
In this article, by James Miller, author of the book Mastering Splunk, we will discuss Splunk lookups and workflows. The topics that will be covered in this article are as follows: The value of a lookup Design lookups File lookups Script lookups (For more resources related to this topic, see here.) Lookups Machines constantly generate data, usually in a raw form that is most efficient for processing by machines, but not easily understood by "human" data consumers. Splunk has the ability to identify unique identifiers and/or result or status codes within the data. This gives you the ability to enhance the readability of the data by adding descriptions or names as new search result fields. These fields contain information from an external source such as a static table (a CSV file) or the dynamic result of a Python command or a Python-based script. Splunk's lookups can use information within returned events or time information to determine how to add other fields from your previously defined external data sources. To illustrate, here is an example of a Splunk static lookup that: Uses the Business Unit value in an event Matches this value with the organization's business unit name in a CSV file Adds the definition to the event (as the Business Unit Name field) So, if you have an event where the Business Unit value is equal to 999999, the lookup will add the Business Unit Name value as Corporate Office to that event. More sophisticated lookups can: Populate a static lookup table from the results of a report. Use a Python script (rather than a lookup table) to define a field. For example, a lookup can use a script to return a server name when given an IP address. Perform a time-based lookup if your lookup table includes a field value that represents time. Let's take a look at an example of a search pipeline that creates a table based on IBM Cognos TM1 file extractions: sourcetype=csv 2014 "Current Forecast" "Direct" "513500" |rename May as "Month" Actual as "Version" "FY 2012" as Year 650693NLR001 as "Business Unit" 100000 as "FCST" "09997_Eliminations Co 2" as "Account" "451200" as "Activity" | eval RFCST= round(FCST) |Table Month, "Business Unit", RFCST The following table shows the results generated:   Now, add the lookup command to our search pipeline to have Splunk convert Business Unit into Business Unit Name: sourcetype=csv 2014 "Current Forecast" "Direct" "513500" |rename May as "Month" Actual as "Version" "FY 2012" as Year 650693NLR001 as "Business Unit" 100000 as "FCST" "09997_Eliminations Co 2" as "Account" "451200"as "Activity" | eval RFCST= round(FCST) |lookup BUtoBUName BU as "Business Unit" OUTPUT BUName as "Business Unit Name" | Table Month, "Business Unit", "Business Unit Name", RFCST The lookup command in our Splunk search pipeline will now add Business Unit Name in the results table:   Configuring a simple field lookup In this section, we will configure a simple Splunk lookup. Defining lookups in Splunk Web You can set up a lookup using the Lookups page (in Splunk Web) or by configuring stanzas in the props.conf and transforms.conf files. Let's take the easier approach first and use the Splunk Web interface. Before we begin, we need to establish our lookup table that will be in the form of an industry standard comma separated file (CSV). Our example is one that converts business unit codes to a more user-friendly business unit name. For example, we have the following information: Business unit code Business unit name 999999 Corporate office VA0133SPS001 South-western VA0133NLR001 North-east 685470NLR001 Mid-west In the events data, only business unit codes are included. In an effort to make our Splunk search results more readable, we want to add the business unit name to our results table. To do this, we've converted our information (shown in the preceding table) to a CSV file (named BUtoBUName.csv):   For this example, we've kept our lookup table simple, but lookup tables (files) can be as complex as you need them to be. They can have numerous fields (columns) in them. A Splunk lookup table has a few requirements, as follows: A table must contain a minimum of two columns Each of the columns in the table can have duplicate values You should use (plain) ASCII text and not non-UTF-8 characters Now, from Splunk Web, we can click on Settings and then select Lookups:   From the Lookups page, we can select Lookup table files:   From the Lookup table files page, we can add our new lookup file (BUtoBUName.csv):   By clicking on the New button, we see the Add new page where we can set up our file by doing the following: Select a Destination app (this is a drop-down list and you should select Search). Enter (or browse to) our file under Upload a lookup file. Provide a Destination filename. Then, we click on Save:   Once you click on Save, you should receive the Successfully saved "BUtoBUName" in search" message:   In the previous screenshot, the lookup file is saved by default as private. You will need to adjust permissions to allow other Splunk users to use it. Going back to the Lookups page, we can select Lookup definitions to see the Lookup definitions page:   In the Lookup definitions page, we can click on New to visit the Add new page (shown in the following screenshot) and set up our definition as follows: Destination app: The lookup will be part of the Splunk search app Name: Our file is BUtoBUName Type: Here, we will select File-based Lookup file: The filename is ButoBUName.csv, which we uploaded without the .csv suffix Again, we should see the Successfully saved "BUtoBUName" in search message:   Now, our lookup is ready to be used: Automatic lookups Rather than having to code for a lookup in each of your Splunk searches, you have the ability to configure automatic lookups for a particular source type. To do this from Splunk Web, we can click on Settings and then select Lookups:   From the Lookups page, click on Automatic lookups:   In the Automatic lookups page, click on New:   In the Add New page, we will fill in the required information to set up our lookup: Destination app: For this field, some options are framework, launcher, learned, search, and splunk_datapreview (for our example, select search). Name: This provide a user-friendly name that describes this automatic lookup. Lookup table: This is the name of the lookup table you defined with a CSV file (discussed earlier in this article). Apply to: This is the type that you want this automatic lookup to apply to. The options are sourcetype, source, or host (I've picked sourcetype). Named: This is the name of the type you picked under Apply to. I want my automatic search to apply for all searches with the sourcetype of csv. Lookup input fields: This is simple in my example. In my lookup table, the field to be searched on will be BU and the = field value will be the field in the event results that I am converting; in my case, it was the field 650693NLR001. Lookup output fields: This will be the field in the lookup table that I am using to convert to, which in my example is BUName and I want to call it Business Unit Name, so this becomes the = field value. Overwrite field values: This is a checkbox where you can tell Splunk to overwrite existing values in your output fields—I checked it. The Add new page The Splunk Add new page (shown in the following screenshot) is where you enter the lookup information (detailed in the previous section):   Once you have entered your automatic lookup information, you can click on Save and you will receive the Successfully saved "Business Unit to Business Unit Name" in search message:   Now, we can use the lookup in a search. For example, you can run a search with sourcetype=csv, as follows: sourcetype=csv 2014 "Current Forecast" "Direct" "513500" |rename May as "Month" Actual as "Version" "FY 2012" as Year 650693NLR001 as "Business Unit" 100000 as "FCST" "09997_Eliminations Co 2"as "Account" "451200" as "Activity" | eval RFCST= round(FCST) |Table "Business Unit", "Business Unit Name", Month, RFCST Notice in the following screenshot that Business Unit Name is converted to the user-friendly values from our lookup table, and we didn't have to add the lookup command to our search pipeline:   Configuration files In addition to using the Splunk web interface, you can define and configure lookups using the following files: props.conf transforms.conf To set up a lookup with these files (rather than using Splunk web), we can perform the following steps: Edit transforms.conf to define the lookup table. The first step is to edit the transforms.conf configuration file to add the new lookup reference. Although the file exists in the Splunk default folder ($SPLUNK_HOME/etc/system/default), you should edit the file in $SPLUNK_HOME/etc/system/local/ or $SPLUNK_HOME/etc/apps/<app_name>/local/ (if the file doesn't exist here, create it). Whenever you edit a Splunk .conf file, always edit a local version, keeping the original (system directory version) intact. In the current version of Splunk, there are two types of lookup tables: static and external. Static lookups use CSV files, and external (which are dynamic) lookups use Python scripting. You have to decide if your lookup will be static (in a file) or dynamic (use script commands). If you are using a file, you'll use filename; if you are going to use a script, you use external_cmd (both will be set in the transforms.conf file). You can also limit the number of matching entries to apply to an event by setting the max_matches option (this tells Splunk to use the first <integer> (in file order) number of entries). I've decided to leave the default for max_matches, so my transforms.conf file looks like the following: [butobugroup]filename = butobugroup.csv This step is optional. Edit props.conf to apply your lookup table automatically. For both static and external lookups, you stipulate the fields you want to match in the configuration file and the output from the lookup table that you defined in your transforms.conf file. It is okay to have multiple field lookups defined in one source lookup definition, but each lookup should have its own unique lookup name; for example, if you have multiple tables, you can name them LOOKUP-table01, LOOKUP-table02, and so on, or something perhaps more easily understood. If you add a lookup to your props.conf file, this lookup is automatically applied to all events from searches that have matching source types (again, as mentioned earlier; if your automatic lookup is very slow, it will also impact the speed of your searches). Restart Splunk to see your changes. Implementing a lookup using configuration files – an example To illustrate the use of configuration files in order to implement an automatic lookup, let's use a simple example. Once again, we want to convert a field from a unique identification code for an organization's business unit to a more user friendly descriptive name called BU Group. What we will do is match the field bu in a lookup table butobugroup.csv with a field in our events. Then, add the bugroup (description) to the returned events. The following shows the contents of the butobugroup.csv file: bu, bugroup 999999, leadership-groupVA0133SPS001, executive-group650914FAC002, technology-group You can put this file into $SPLUNK_HOME/etc/apps/<app_name>/lookups/ and carry out the following steps: Put the butobugroup.csv file into $SPLUNK_HOME/etc/apps/search/lookups/, since we are using the search app. As we mentioned earlier, we edit the transforms.conf file located at either $SPLUNK_HOME/etc/system/local/ or $SPLUNK_HOME/etc/apps/<app_name>/local/. We add the following two lines: [butobugroup]filename = butobugroup.csv Next, as mentioned earlier in this article, we edit the props.conf file located at either $SPLUNK_HOME/etc/system/local/ or $SPLUNK_HOME/etc/apps/<app_name>/local/. Here, we add the following two lines: [csv]LOOKUP-check = butobugroup bu AS 650693NLR001 OUTPUT bugroup Restart the Splunk server. You can (assuming you are logged in as an admin or have admin privileges) restart the Splunk server through the web interface by going to Settings, then select System and finally Server controls. Now, you can run a search for sourcetype=csv (as shown here): sourcetype=csv 2014 "Current Forecast" "Direct" "513500" |rename May as "Month" ,650693NLR001 as "Business Unit" 100000 as "FCST"| eval RFCST= round(FCST) |Table "Business Unit", "Business Unit Name", bugroup, Month, RFCST You will see that the field bugroup can be returned as part of your event results:   Populating lookup tables Of course, you can create CSV files from external systems (or, perhaps even manually?), but from time to time, you might have the opportunity to create lookup CSV files (tables) from event data using Splunk. A handy command to accomplish this is outputcsv (which is covered in detail later in this article). The following is a simple example of creating a CSV file from Splunk event data that can be used for a lookup table: sourcetype=csv "Current Forecast" "Direct" | rename 650693NLR001 as "Business Unit" | Table "Business Unit", "Business Unit Name", bugroup | outputcsv splunk_master The results are shown in the following screeshot:   Of course, the output table isn't quite usable, since the results have duplicates. Therefore, we can rewrite the Splunk search pipeline introducing the dedup command (as shown here): sourcetype=csv   "Current Forecast" "Direct"   | rename 650693NLR001 as "Business Unit" | dedup "Business Unit" | Table "Business Unit", "Business Unit Name", bugroup | outputcsv splunk_master Then, we can examine the results (now with more desirable results):   Handling duplicates with dedup This command allows us to set the number of duplicate events to be kept based on the values of a field (in other words, we can use this command to drop duplicates from our event results for a selected field). The event returned for the dedup field will be the first event found (if you provide a number directly after the dedup command, it will be interpreted as the number of duplicate events to keep; if you don't specify a number, dedup keeps only the first occurring event and removes all consecutive duplicates). The dedup command also lets you sort by field or list of fields. This will remove all the duplicates and then sort the results based on the specified sort-by field. Adding a sort in conjunction with the dedup command can affect the performance as Splunk performs the dedup operation and then sorts the results as a final step. Here is a search command using dedup: sourcetype=csv   "Current Forecast" "Direct"   | rename 650693NLR001 as "Business Unit" | dedup "Business Unit" sortby bugroup | Table "Business Unit", "Business Unit Name", bugroup | outputcsv splunk_master The result of the preceding command is shown in the following screenshot:   Now, we have our CSV lookup file (outputcsv splunk_master) generated and ready to be used:   Look for your generated output file in $SPLUNK_HOME/var/run/splunk. Dynamic lookups With a Splunk static lookup, your search reads through a file (a table) that was created or updated prior to executing the search. With dynamic lookups, the file is created at the time the search executes. This is possible because Splunk has the ability to execute an external command or script as part of your Splunk search. At the time of writing this book, Splunk only directly supports Python scripts for external lookups. If you are not familiar with Python, its implementation began in 1989 and is a widely used general-purpose, high-level programming language, which is often used as a scripting language (but is also used in a wide range of non-scripting contexts). Keep in mind that any external resources (such as a file) or scripts that you want to use with your lookup will need to be copied to a location where Splunk can find it. These locations are: $SPLUNK_HOME/etc/apps/<app_name>/bin $SPLUNK_HOME/etc/searchscripts The following sections describe the process of using the dynamic lookup example script that ships with Splunk (external_lookup.py). Using Splunk Web Just like with static lookups, Splunk makes it easy to define a dynamic or external lookup using the Splunk web interface. First, click on Settings and then select Lookups:   On the Lookups page, we can select Lookup table files to define a CSV file that contains the input file for our Python script. In the Add new page, we enter the following information: Destination app: For this field, select Search Upload a lookup file: Here, you can browse to the filename (my filename is dnsLookup.csv) Destination filename: Here, enter dnslookup The Add new page is shown in the following screenshot:   Now, click on Save. The lookup file (shown in the following screenshot) is a text CSV file that needs to (at a minimum) contain the two field names that the Python (py) script accepts as arguments, in this case, host and ip. As mentioned earlier, this file needs to be copied to $SPLUNK_HOME/etc/apps/<app_name>/bin.   Next, from the Lookups page, select Lookup definitions and then click on New. This is where you define your external lookup. Enter the following information: Type: For this, select External (as this lookup will run an external script) Command: For this, enter external_lookup.py host ip (this is the name of the py script and its two arguments) Supported fields: For this, enter host, ip (this indicates the two script input field names) The following screenshot describes a new lookup definition:   Now, click on Save. Using configuration files instead of Splunk Web Again, just like with static lookups in Splunk, dynamic lookups can also be configured in the Splunk transforms.conf file: [myLookup]external_cmd = external_lookup.py host ipexternal_type = pythonfields_list = host, ipmax_matches = 200 Let's learn more about the terms here: [myLookup]: This is the report stanza. external_cmd: This is the actual runtime command definition. Here, it executes the Python (py) script external_lookup, which requires two arguments (or parameters), host and ip. external_type (optional): This indicates that this is a Python script. Although this is an optional entry in the transform.conf file, it's a good habit to include this for readability and support. fields_list: This lists all the fields supported by the external command or script, delimited by a comma and space. The next step is to modify the props.conf file, as follows: [mylookup]LOOKUP-rdns = dnslookup host ip OUTPUT ip After updating the Splunk configuration files, you will need to restart Splunk. External lookups The external lookup example given uses a Python (py) script named external_lookup.py, which is a DNS lookup script that can return an IP address for a given host name or a host name for a provided IP address. Explanation The lookup table field in this example is named ip, so Splunk will mine all of the IP addresses found in the indexed logs' events and add the values of ip from the lookup table into the ip field in the search events. We can notice the following: If you look at the py script, you will notice that the example uses an MS Windows supported socket.gethostbyname_ex(host) function The host field has the same name in the lookup table and the events, so you don't need to do anything else Consider the following search command: sourcetype=tm1* | lookup dnslookup host | table host, ip When you run this command, Splunk uses the lookup table to pass the values for the host field as a CSV file (the text CSV file we looked at earlier) into the external command script. The py script then outputs the results (with both the host and ip fields populated) and returns it to Splunk, which populates the ip field in a result table:   Output of the py script with both the host and ip fields populated Time-based lookups If your lookup table has a field value that represents time, you can use the time field to set up a Splunk fields lookup. As mentioned earlier, the Splunk transforms.conf file can be modified to add a lookup stanza. For example, the following screenshot shows a file named MasteringDCHP.csv:   You can add the following code to the transforms.conf file: [MasteringDCHP]filename = MasteringDCHP.csvtime_field = TimeStamptime_format = %d/%m/%y %H:%M:%S $pmax_offset_secs = <integer>min_offset_secs = <integer> The file parameters are defined as follows: [MasteringDCHP]: This is the report stanza filename: This is the name of the CSV file to be used as the lookup table time_field: This is the field in the file that contains the time information and is to be used as the timestamp time_format: This indicates what format the time field is in max_offset_secs and min_offset_secs: This indicates min/max amount of offset time for an event to occur after a lookup entry Be careful with the preceding values; the offset relates to the timestamp in your lookup (CSV) file. Setting a tight (small) offset range might reduce the effectiveness of your lookup results! The last step will be to restart Splunk. An easier way to create a time-based lookup Again, it's a lot easier to use the Splunk Web interface to set up our lookup. Here is the step-by-step process: From Settings, select Lookups, and then Lookup table files: In the Lookup table files page, click on New, configure our lookup file, and then click on Save: You should receive the Successfully saved "MasterDHCP" in search message: Next, select Lookup definitions and from this page, click on New: In the Add new page, we define our lookup table with the following information: Destination app: For this, select search from the drop-down list Name: For this, enter MasterDHCP (this is the name you'll use in your lookup) Type: For this, select File-based (as this lookup table definition is a CSV file) Lookup file: For this, select the name of the file to be used from the drop-down list (ours is MasteringDCHP) Configure time-based lookup: Check this checkbox Name of time field: For this, enter TimeStamp (this is the field name in our file that contains the time information) Time format: For this, enter the string to describe to Splunk the format of our time field (our field uses this format: %d%m%y %H%M%S) You can leave the rest blank and click on Save. You should receive the Successfully saved "MasterDHCP" in search message: Now, we are ready to try our search: sourcetype=dh* | Lookup MasterDHCP IP as "IP" | table DHCPTimeStamp, IP, UserId | sort UserId The following screenshot shows the output:   Seeing double? Lookup table definitions are indicated with the attribute LOOKUP-<class> in the Splunk configuration file, props.conf, or in the web interface under Settings | Lookups | Lookup definitions. If you use the Splunk Web interface (which we've demonstrated throughout this article) to set up or define your lookup table definitions, Splunk will prevent you from creating duplicate table names, as shown in the following screenshot:   However, if you define your lookups using the configuration settings, it is important to try and keep your table definition names unique. If you do give the same name to multiple lookups, the following rules apply: If you have defined lookups with the same stanza (that is, using the same host, source, or source type), the first defined lookup in the configuration file wins and overrides all others. If lookups have different stanzas but overlapping events, the following logic is used by Splunk: Events that match the host get the host lookup Events that match the sourcetype get the sourcetype lookup Events that match both only get the host lookup It is a proven practice recommendation to make sure that all of your lookup stanzas have unique names. Command roundup This section lists several important Splunk commands you will use when working with lookups. The lookup command The Splunk lookup command is used to manually invoke field lookups using a Splunk lookup table that is previously defined. You can use Splunk Web (or the transforms.conf file) to define your lookups. If you do not specify OUTPUT or OUTPUTNEW, all fields in the lookup table (excluding the lookup match field) will be used by Splunk as output fields. Conversely, if OUTPUT is specified, the output lookup fields will overwrite existing fields and if OUTPUTNEW is specified, the lookup will not be performed for events in which the output fields already exist. For example, if you have a lookup table specified as iptousername with (at least) two fields, IP and UserId, for each event, Splunk will look up the value of the field IP in the table and for any entries that match, the value of the UserId field in the lookup table will be written to the field user_name in the event. The query is as follows: ... Lookup iptousernameIP as "IP" output UserId as user_name Always strive to perform lookups after any reporting commands in your search pipeline, so that the lookup only needs to match the results of the reporting command and not every individual event. The inputlookup and outputlookup commands The inputlookup command allows you to load search results from a specified static lookup table. It reads in a specified CSV filename (or a table name as specified by the stanza name in transforms.conf). If the append=t (that is, true) command is added, the data from the lookup file is appended to the current set of results (instead of replacing it). The outputlookup command then lets us write the results' events to a specified static lookup table (as long as this output lookup table is defined). So, here is an example of reading in the MasterDHCP lookup table (as specified in transforms.conf) and writing these event results to the lookup table definition NewMasterDHCP: | inputlookup MasterDHCP | outputlookup NewMasterDHCP After running the preceding command, we can see the following output:   Note that we can add the append=t command to the search in the following fashion: | inputlookup MasterDHCP.csv | inputlookup NewMasterDHCP.csv append=t | The inputcsv and outputcsv commands The inputcsv command is similar to the inputlookup command; in this, it loads search results, but this command loads from a specified CSV file. The filename must refer to a relative path in $SPLUNK_HOME/var/run/splunk and if the specified file does not exist and the filename did not have an extension, then a filename with a .csv extension is assumed. The outputcsv command lets us write our result events to a CSV file. Here is an example where we read in a CSV file named splunk_master.csv, search for the text phrase FPM, and then write any matching events to a CSV file named FPMBU.csv: | inputcsv splunk_master.csv | search "Business Unit Name"="FPM" | outputcsv FPMBU.csv The following screenshot shows the results from the preceding search command:   The following screenshot shows the resulting file generated as a result of the preceding command:   Here is another example where we read in the same CSV file (splunk_master.csv) and write out only events from 51 to 500: | inputcsv splunk_master start=50 max=500 Events are numbered starting with zero as the first entry (rather than 1). Summary In this article, we defined Splunk lookups and discussed their value. We also went through the two types of lookups, static and dynamic, and saw detailed, working examples of each. Various Splunk commands typically used with the lookup functionality were also presented. Resources for Article: Further resources on this subject: Working with Apps in Splunk [article] Processing Tweets with Apache Hive [article] Indexes [article]
Read more
  • 0
  • 0
  • 8397

article-image-introduction-logging-tomcat-7
Packt
21 Mar 2012
9 min read
Save for later

Introduction to Logging in Tomcat 7

Packt
21 Mar 2012
9 min read
(For more resources on Apache, see here.) JULI Previous versions of Tomcat (till 5.x) use Apache common logging services for generating logs. A major disadvantage with this logging mechanism is that it can handle only single JVM configuration and makes it difficult to configure separate logging for each class loader for independent application. In order to resolve this issue, Tomcat developers have introduced a separate API for Tomcat 6 version, that comes with the capability of capturing each class loader activity in the Tomcat logs. It is based on java.util.logging framework. By default, Tomcat 7 uses its own Java logging API to implement logging services. This is also called as JULI. This API can be found in TOMCAT_HOME/bin of the Tomcat 7 directory structures (tomcat-juli.jar). The following screenshot shows the directory structure of the bin directory where tomcat-juli.jar is placed. JULI also provides the feature for custom logging for each web application, and it also supports private per-application logging configurations. With the enhanced feature of separate class loader logging, it also helps in detecting memory issues while unloading the classes at runtime. For more information on JULI and the class loading issue, please refer to http://tomcat.apache.org/tomcat-7.0-doc/logging.html and http://tomcat.apache.org/tomcat-7.0-doc/class-loader-howto.html respectively. Loggers, appenders, and layouts There are some important components of logging which we use at the time of implementing the logging mechanism for applications. Each term has its individual importance in tracking the events of the application. Let's discuss each term individually to find out their usage: Loggers:It can be defined as the logical name for the log file. This logical name is written in the application code. We can configure an independent logger for each application. Appenders: The process of generation of logs are handled by appenders. There are many types of appenders, such as FileAppender, ConsoleAppender, SocketAppender, and so on, which are available in log4j. The following are some examples of appenders for log4j: log4j.appender.CATALINA=org.apache.log4j.DailyRollingFileAppender log4j.appender.CATALINA.File=${catalina.base}/logs/catalina.out log4j.appender.CATALINA.Append=true log4j.appender.CATALINA.Encoding=UTF-8 The previous four lines of appenders define the DailyRollingFileAppender in log4j, where the filename is catalina.out . These logs will have UTF-8 encoding enabled. If log4j.appender.CATALINA.append=false, then logs will not get updated in the log files. # Roll-over the log once per day log4j.appender.CATALINA.DatePattern='.'dd-MM-yyyy'.log' log4j.appender.CATALINA.layout = org.apache.log4j.PatternLayout log4j.appender.CATALINA.layout.ConversionPattern = %d [%t] %-5p %c- %m%n The previous three lines of code show the roll-over of log once per day. Layout: It is defined as the format of logs displayed in the log file. The appender uses layout to format the log files (also called as patterns). The highlighted code shows the pattern for access logs: <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log." suffix=".txt" pattern="%h %l %u %t &quot;%r&quot; %s %b" resolveHosts="false"/> Loggers, appenders, and layouts together help the developer to capture the log message for the application event. Types of logging in Tomcat 7 We can enable logging in Tomcat 7 in different ways based on the requirement. There are a total of five types of logging that we can configure in Tomcat, such as application, server, console, and so on. The following figure shows the different types of logging for Tomcat 7. These methods are used in combination with each other based on environment needs. For example, if you have issues where Tomcat services are not displayed, then console logs are very helpful to identify the issue, as we can verify the real-time boot sequence. Let's discuss each logging method briefly. Application log These logs are used to capture the application event while running the application transaction. These logs are very useful in order to identify the application level issues. For example, suppose your application performance is slow on a particular transition, then the details of that transition can only be traced in application log. The biggest advantage of application logs is we can configure separate log levels and log files for each application, making it very easy for the administrators to troubleshoot the application. Log4j is used in 90 percent of the cases for application log generation. Server log Server logs are identical to console logs. The only advantage of server logs is that they can be retrieved anytime but console logs are not available after we log out from the console. Console log This log gives you the complete information of Tomcat 7 startup and loader sequence. The log file is named as catalina.out and is found in TOMCAT_HOME/logs. This log file is very useful in checking the application deployment and server startup testing for any environment. This log is configured in the Tomcat file catalina.sh, which can be found in TOMCAT_HOME/bin. The previous screenshot shows the definition for Tomcat logging. By default, the console logs are configured as INFO mode. There are different levels of logging in Tomcat such as WARNING, INFORMATION, CONFIG, and FINE. The previous screenshot shows the Tomcat log file location, after the start of Tomcat services. The previous screenshot shows the output of the catalina.out file, where Tomcat services are started in 1903 ms. Access log Access logs are customized logs, which give information about the following: Who has accessed the application What components of the application are accessed Source IP and so on These logs play a vital role in traffic analysis of many applications to analyze the bandwidth requirement and also helps in troubleshooting the application under heavy load. These logs are configured in server.xml in TOMCAT_HOME/conf. The following screenshot shows the definition of access logs. You can customize them according to the environment and your auditing requirement. Let's discuss the pattern format of the access logs and understand how we can customize the logging format: <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="localhost_access_log." suffix=".txt" pattern="%h %l %u %t &quot;%r&quot; %s %b" resolveHosts="false"/> Class Name: This parameter defines the class name used for generation of logs. By default, Apache Tomcat 7 uses the org.apache.catalina.valves.AccessLogValve class for access logs. Directory: This parameter defines the directory location for the log file. All the log files are generated in the log directory—TOMCAT_HOME/logs—but we can customize the log location based on our environment setup and then update the directory path in the definition of access logs. Prefix: This parameter defines the prefix of the access log filename, that is, by default, access log files are generated by the name localhost_access_log.yy-mm-dd.txt. Suffix: This parameter defines the file extension of the log file. Currently it is in .txt format. Pattern: This parameter defines the format of the log file. The pattern is a combination of values defined by the administrator, for example, %h = remote host address. The following screenshot shows the default log format for Tomcat 7. Access logs show the remote host address, date/time of request, method used for response, URI mapping, and HTTP status code. In case you have installed the web traffic analysis tool for application, then you have to change the access logs to a different format. Host manager These logs define the activity performed using Tomcat Manager, such as various tasks performed, status of application, deployment of application, and lifecycle of Tomcat. These configurations are done on the logging.properties, which can be found in TOMCAT_HOME/conf. The previous screenshot shows the definition of host, manager, and host-manager details. If you see the definitions, it defines the log location, log level, and prefix of the filename. In logging.properties, we are defining file handlers and appenders using JULI. The log file for manager looks similar to the following: I28 Jun, 2011 3:36:23 AM org.apache.catalina.core.ApplicationContext log INFO: HTMLManager: list: Listing contexts for virtual host 'localhost' 28 Jun, 2011 3:37:13 AM org.apache.catalina.core.ApplicationContext log INFO: HTMLManager: list: Listing contexts for virtual host 'localhost' 28 Jun, 2011 3:37:42 AM org.apache.catalina.core.ApplicationContext log INFO: HTMLManager: undeploy: Undeploying web application at '/sample' 28 Jun, 2011 3:37:43 AM org.apache.catalina.core.ApplicationContext log INFO: HTMLManager: list: Listing contexts for virtual host 'localhost' 28 Jun, 2011 3:42:59 AM org.apache.catalina.core.ApplicationContext log INFO: HTMLManager: list: Listing contexts for virtual host 'localhost' 28 Jun, 2011 3:43:01 AM org.apache.catalina.core.ApplicationContext log INFO: HTMLManager: list: Listing contexts for virtual host 'localhost' 28 Jun, 2011 3:53:44 AM org.apache.catalina.core.ApplicationContext log INFO: HTMLManager: list: Listing contexts for virtual host 'localhost' Types of log levels in Tomcat 7 There are seven levels defined for Tomcat logging services (JULI). They can be set based on the application requirement. The following figure shows the sequence of log levels for JULI: Every log level in JULI had its own functionality. The following table shows the functionality of each log level in JULI: Log level Description SEVERE(highest) Captures exception and Error WARNING Warning messages INFO Informational message, related to server activity CONFIG Configuration message FINE Detailed activity of server transaction (similar to debug) FINER More detailed logs than FINE FINEST(least) Entire flow of events (similar to trace) For example, let's take an appender from logging.properties and find out the log level used; the first log appender for localhost is using FINE as the log level, as shown in the following code snippet: localhost.org.apache.juli.FileHandler.level = FINE localhost.org.apache.juli.FileHandler.directory = ${catalina.base}/logs localhost.org.apache.juli.FileHandler.prefix = localhost. The following code shows the default file handler configuration for logging in Tomcat 7 using JULI. The properties are mentioned and log levels are highlighted: ############################################################ # Facility specific properties. # Provides extra control for each logger. ############################################################ org.apache.catalina.core.ContainerBase.[Catalina].[localhost].level = INFO org.apache.catalina.core.ContainerBase.[Catalina].[localhost].handlers = 2localhost.org.apache.juli.FileHandler org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager] .level = INFO org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/manager] .handlers = 3manager.org.apache.juli.FileHandler org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host- manager].level = INFO org.apache.catalina.core.ContainerBase.[Catalina].[localhost].[/host- manager].handlers = 4host-manager.org.apache.juli.FileHandler
Read more
  • 0
  • 0
  • 8394

article-image-writing-custom-spring-boot-starters
Packt
16 Sep 2015
10 min read
Save for later

Writing Custom Spring Boot Starters

Packt
16 Sep 2015
10 min read
 In this article by Alex Antonov, author of the book Spring Boot Cookbook, we will cover the following topics: Understanding Spring Boot autoconfiguration Creating a custom Spring Boot autoconfiguration starter (For more resources related to this topic, see here.) Introduction Its time to take a look behind the scenes and find out the magic behind the Spring Boot autoconfiguration and write some starters of our own as well. This is a very useful capability to possess, especially for large software enterprises where the presence of a proprietary code is inevitable and it is very helpful to be able to create internal custom starters that would automatically add some of the configuration or functionalities to the applications. Some likely candidates can be custom configuration systems, libraries, and configurations that deal with connecting to databases, using custom connection pools, http clients, servers, and so on. We will go through the internals of Spring Boot autoconfiguration, take a look at how new starters are created, explore conditional initialization and wiring of beans based on various rules, and see that annotations can be a powerful tool, which provides the consumers of the starters more control over dictating what configurations should be used and where. Understanding Spring Boot autoconfiguration Spring Boot has a lot of power when it comes to bootstrapping an application and configuring it with exactly the things that are needed, all without much of the glue code that is required of us, the developers. The secret behind this power actually comes from Spring itself or rather from the Java Configuration functionality that it provides. As we add more starters as dependencies, more and more classes will appear in our classpath. Spring Boot detects the presence or absence of specific classes and based on this information, makes some decisions, which are fairly complicated at times, and automatically creates and wires the necessary beans to the application context. Sounds simple, right? How to do it… Conveniently, Spring Boot provides us with an ability to get the AUTO-CONFIGURATION REPORT by simply starting the application with the debug flag. This can be passed to the application either as an environment variable, DEBUG, as a system property, -Ddebug, or as an application property, --debug. Start the application by running DEBUG=true ./gradlew clean bootRun. Now, if you look at the console logs, you will see a lot more information printed there that is marked with the DEBUG level log. At the end of the startup log sequence, we will see the AUTO-CONFIGURATION REPORT as follows: ========================= AUTO-CONFIGURATION REPORT ========================= Positive matches: ----------------- … DataSourceAutoConfiguration - @ConditionalOnClass classes found: javax.sql.DataSource,org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseType (OnClassCondition) … Negative matches: ----------------- … GsonAutoConfiguration - required @ConditionalOnClass classes not found: com.google.gson.Gson (OnClassCondition) … How it works… As you can see, the amount of information that is printed in the debug mode can be somewhat overwhelming; so I've selected only one example of positive and negative matches each. For each line of the report, Spring Boot tells us why certain configurations have been selected to be included, what they have been positively matched on, or, for the negative matches, what was missing that prevented a particular configuration to be included in the mix. Let's look at the positive match for DataSourceAutoConfiguration: The @ConditionalOnClass classes found tells us that Spring Boot has detected the presence of a particular class, specifically two classes in our case: javax.sql.DataSource and org.springframework.jdbc.datasource.embedded.EmbeddedDatabaseType. The OnClassCondition indicates the kind of matching that was used. This is supported by the @ConditionalOnClass and @ConditionalOnMissingClass annotations. While OnClassCondition is the most common kind of detection, Spring Boot also uses many other conditions. For example, OnBeanCondition is used to check the presence or absence of specific bean instances, OnPropertyCondition is used to check the presence, absence, or a specific value of a property as well as any number of the custom conditions that can be defined using the @Conditional annotation and Condition interface implementations. The negative matches show us a list of configurations that Spring Boot has evaluated, which means that they do exist in the classpath and were scanned by Spring Boot but didn't pass the conditions required for their inclusion. GsonAutoConfiguration, while available in the classpath as it is a part of the imported spring-boot-autoconfigure artifact, was not included because the required com.google.gson.Gson class was not detected as present in the classpath, thus failing the OnClassCondition. The implementation of the GsonAutoConfiguration file looks as follows: @Configuration @ConditionalOnClass(Gson.class) public class GsonAutoConfiguration { @Bean @ConditionalOnMissingBean public Gson gson() { return new Gson(); } } After looking at the code, it is very easy to make the connection between the conditional annotations and report information that is provided by Spring Boot at the start time. Creating a custom Spring Boot autoconfiguration starter We have a high-level idea of the process by which Spring Boot decides which configurations to include in the formation of the application context. Now, let's take a stab at creating our own Spring Boot starter artifact, which we can include as an autoconfigurable dependency in our build. Let's build a simple starter that will create another CommandLineRunner that will take the collection of all the Repository instances and print out the count of the total entries for each. We will start by adding a child Gradle project to our existing project that will house the codebase for the starter artifact. We will call it db-count-starter. How to do it… We will start by creating a new directory named db-count-starter in the root of our project. As our project has now become what is known as a multiproject build, we will need to create a settings.gradle configuration file in the root of our project with the following content: include 'db-count-starter' We should also create a separate build.gradle configuration file for our subproject in the db-count-starter directory in the root of our project with the following content: apply plugin: 'java' repositories { mavenCentral() maven { url "https://repo.spring.io/snapshot" } maven { url "https://repo.spring.io/milestone" } } dependencies { compile("org.springframework.boot:spring- boot:1.2.3.RELEASE") compile("org.springframework.data:spring-data- commons:1.9.2.RELEASE") } Now we are ready to start coding. So, the first thing is to create the directory structure, src/main/java/org/test/bookpubstarter/dbcount, in the db-count-starter directory in the root of our project. In the newly created directory, let's add our implementation of the CommandLineRunner file named DbCountRunner.java with the following content: public class DbCountRunner implements CommandLineRunner { protected final Log logger = LogFactory.getLog(getClass()); private Collection<CrudRepository> repositories; public DbCountRunner(Collection<CrudRepository> repositories) { this.repositories = repositories; } @Override public void run(String... args) throws Exception { repositories.forEach(crudRepository -> logger.info(String.format("%s has %s entries", getRepositoryName(crudRepository.getClass()), crudRepository.count()))); } private static String getRepositoryName(Class crudRepositoryClass) { for(Class repositoryInterface : crudRepositoryClass.getInterfaces()) { if (repositoryInterface.getName(). startsWith("org.test.bookpub.repository")) { return repositoryInterface.getSimpleName(); } } return "UnknownRepository"; } } With the actual implementation of DbCountRunner in place, we will now need to create the configuration object that will declaratively create an instance during the configuration phase. So, let's create a new class file called DbCountAutoConfiguration.java with the following content: @Configuration public class DbCountAutoConfiguration { @Bean public DbCountRunner dbCountRunner(Collection<CrudRepository> repositories) { return new DbCountRunner(repositories); } } We will also need to tell Spring Boot that our newly created JAR artifact contains the autoconfiguration classes. For this, we will need to create a resources/META-INF directory in the db-count-starter/src/main directory in the root of our project. In this newly created directory, we will place the file named spring.factories with the following content: org.springframework.boot.autoconfigure.EnableAutoConfiguration=org.test.bookpubstarter.dbcount.DbCountAutoConfiguration For the purpose of our demo, we will add the dependency to our starter artifact in the main project's build.gradle by adding the following entry in the dependencies section: compile project(':db-count-starter') Start the application by running ./gradlew clean bootRun. Once the application is complied and has started, we should see the following in the console logs: 2015-04-05 INFO org.test.bookpub.StartupRunner : Welcome to the Book Catalog System! 2015-04-05 INFO o.t.b.dbcount.DbCountRunner : AuthorRepository has 1 entries 2015-04-05 INFO o.t.b.dbcount.DbCountRunner : PublisherRepository has 1 entries 2015-04-05 INFO o.t.b.dbcount.DbCountRunner : BookRepository has 1 entries 2015-04-05 INFO o.t.b.dbcount.DbCountRunner : ReviewerRepository has 0 entries 2015-04-05 INFO org.test.bookpub.BookPubApplication : Started BookPubApplication in 8.528 seconds (JVM running for 9.002) 2015-04-05 INFO org.test.bookpub.StartupRunner           : Number of books: 1 How it works… Congratulations! You have now built your very own Spring Boot autoconfiguration starter. First, let's quickly walk through the changes that we made to our Gradle build configuration and then we will examine the starter setup in detail. As the Spring Boot starter is a separate, independent artifact, just adding more classes to our existing project source tree would not really demonstrate much. To make this separate artifact, we had a few choices: making a separate Gradle configuration in our existing project or creating a completely separate project altogether. The most ideal solution, however, was to just convert our build to Gradle Multi-Project Build by adding a nested project directory and subproject dependency to build.gradle of the root project. By doing this, Gradle actually creates a separate artifact JAR for us but we don't have to publish it anywhere, only include it as a compile project(':db-count-starter') dependency. For more information about Gradle multi-project builds, you can check out the manual at http://gradle.org/docs/current/userguide/multi_project_builds.html. Spring Boot Auto-Configuration Starter is nothing more than a regular Spring Java Configuration class annotated with the @Configuration annotation and the presence of spring.factories in the classpath in the META-INF directory with the appropriate configuration entries. During the application startup, Spring Boot uses SpringFactoriesLoader, which is a part of Spring Core, in order to get a list of the Spring Java Configurations that are configured for the org.springframework.boot.autoconfigure.EnableAutoConfiguration property key. Under the hood, this call collects all the spring.factories files located in the META-INF directory from all the jars or other entries in the classpath and builds a composite list to be added as application context configurations. In addition to the EnableAutoConfiguration key, we can declare the following automatically initializable other startup implementations in a similar fashion: org.springframework.context.ApplicationContextInitializer org.springframework.context.ApplicationListener org.springframework.boot.SpringApplicationRunListener org.springframework.boot.env.PropertySourceLoader org.springframework.boot.autoconfigure.template.TemplateAvailabilityProvider org.springframework.test.contex.TestExecutionListener Ironically enough, a Spring Boot Starter does not need to depend on the Spring Boot library as its compile time dependency. If we look at the list of class imports in the DbCountAutoConfiguration class, we will not see anything from the org.springframework.boot package. The only reason that we have a dependency declared on Spring Boot is because our implementation of DbCountRunner implements the org.springframework.boot.CommandLineRunner interface. Summary Resources for Article: Further resources on this subject: Welcome to the Spring Framework[article] Time Travelling with Spring[article] Aggregators, File exchange Over FTP/FTPS, Social Integration, and Enterprise Messaging[article]
Read more
  • 0
  • 0
  • 8373
article-image-emotional-ai-detecting-facial-expressions-and-emotions-using-coreml-tutorial
Savia Lobo
14 Sep 2018
11 min read
Save for later

Emotional AI: Detecting facial expressions and emotions using CoreML [Tutorial]

Savia Lobo
14 Sep 2018
11 min read
Recently we see computers allow natural forms of interaction and are becoming more ubiquitous, more capable, and more ingrained in our daily lives. They are becoming less like heartless dumb tools and more like friends, able to entertain us, look out for us, and assist us with our work. This article is an excerpt taken from the book Machine Learning with Core ML authored by Joshua Newnham. With this shift comes a need for computers to be able to understand our emotional state. For example, you don't want your social robot cracking a joke after you arrive back from work having lost your job (to an AI bot!). This is a field of computer science known as affective computing (also referred to as artificial emotional intelligence or emotional AI), a field that studies systems that can recognize, interpret, process, and simulate human emotions. The first stage of this is being able to recognize the emotional state. In this article, we will be creating a model that can detect the exact face expression or emotion using CoreML. Input data and preprocessing We will implement the preprocessing functionality required to transform images into something the model is expecting. We will build up this functionality in a playground project before migrating it across to our project in the next section. If you haven't done so already, pull down the latest code from the accompanying repository: https://github.com/packtpublishing/machine-learning-with-core-ml. Once downloaded, navigate to the directory Chapter4/Start/ and open the Playground project ExploringExpressionRecognition.playground. Once loaded, you will see the playground for this extract, as shown in the following screenshot: Before starting, to avoid looking at images of me, please replace the test images with either personal photos of your own or royalty free images from the internet, ideally a set expressing a range of emotions. Along with the test images, this playground includes a compiled Core ML model (we introduced it in the previous image) with its generated set of wrappers for inputs, outputs, and the model itself. Also included are some extensions for UIImage, UIImageView, CGImagePropertyOrientation, and an empty CIImage extension, to which we will return later in the extract. The others provide utility functions to help us visualize the images as we work through this playground. When developing machine learning applications, you have two broad paths. The first, which is becoming increasingly popular, is to use an end-to-end machine learning model capable of just being fed the raw input and producing adequate results. One particular field that has had great success with end-to-end models is speech recognition. Prior to end-to-end deep learning, speech recognition systems were made up of many smaller modules, each one focusing on extracting specific pieces of data to feed into the next module, which was typically manually engineered. Modern speech recognition systems use end-to-end models that take the raw input and output the result. Both of the described approaches can been seen in the following diagram: Obviously, this approach is not constrained to speech recognition and we have seen it applied to image recognition tasks, too, along with many others. But there are two things that make this particular case different; the first is that we can simplify the problem by first extracting the face. This means our model has less features to learn and offers a smaller, more specialized model that we can tune. The second thing, which is no doubt obvious, is that our training data consisted of only faces and not natural images. So, we have no other choice but to run our data through two models, the first to extract faces and the second to perform expression recognition on the extracted faces, as shown in this diagram: Luckily for us, Apple has mostly taken care of our first task of detecting faces through the Vision framework it released with iOS 11. The Vision framework provides performant image analysis and computer vision tools, exposing them through a simple API. This allows for face detection, feature detection and tracking, and classification of scenes in images and video. The latter (expression recognition) is something we will take care of using the Core ML model introduced earlier. Prior to the introduction of the Vision framework, face detection would typically be performed using the Core Image filter. Going back further, you had to use something like OpenCV. You can learn more about Core Image here: https://developer.apple.com/library/content/documentation/GraphicsImaging/Conceptual/CoreImaging/ci_detect_faces/ci_detect_faces.html. Now that we have got a bird's-eye view of the work that needs to be done, let's turn our attention to the editor and start putting all of this together. Start by loading the images; add the following snippet to your playground: var images = [UIImage]() for i in 1...3{ guard let image = UIImage(named:"images/joshua_newnham_\(i).jpg") else{ fatalError("Failed to extract features") } images.append(image) } let faceIdx = 0 let imageView = UIImageView(image: images[faceIdx]) imageView.contentMode = .scaleAspectFit In the preceding snippet, we are simply loading each of the images we have included in our resources' Images folder and adding them to an array we can access conveniently throughout the playground. Once all the images are loaded, we set the constant faceIdx, which will ensure that we access the same images throughout our experiments. Finally, we create an ImageView to easily preview it. Once it has finished running, click on the eye icon in the right-hand panel to preview the loaded image, as shown in the following screenshot: Next, we will take advantage of the functionality available in the Vision framework to detect faces. The typical flow when working with the Vision framework is defining a request, which determines what analysis you want to perform, and defining the handler, which will be responsible for executing the request and providing means of obtaining the results (either through delegation or explicitly queried). The result of the analysis is a collection of observations that you need to cast into the appropriate observation type; concrete examples of each of these can be seen here: As illustrated in the preceding diagram, the request determines what type of image analysis will be performed; the handler, using a request or multiple requests and an image, performs the actual analysis and generates the results (also known as observations). These are accessible via a property or delegate if one has been assigned. The type of observation is dependent on the request performed; it's worth highlighting that the Vision framework is tightly integrated into Core ML and provides another layer of abstraction and uniformity between you and the data and process. For example, using a classification Core ML model would return an observation of type VNClassificationObservation. This layer of abstraction not only simplifies things but also provides a consistent way of working with machine learning models. In the previous figure, we showed a request handler specifically for static images. Vision also provides a specialized request handler for handling sequences of images, which is more appropriate when dealing with requests such as tracking. The following diagram illustrates some concrete examples of the types of requests and observations applicable to this use case: So, when do you use VNImageRequestHandler and VNSequenceRequestHandler? Though the names provide clues as to when one should be used over the other, it's worth outlining some differences. The image request handler is for interactive exploration of an image; it holds a reference to the image for its life cycle and allows optimizations of various request types. The sequence request handler is more appropriate for performing tasks such as tracking and does not optimize for multiple requests on an image. Let's see how this all looks in code; add the following snippet to your playground: let faceDetectionRequest = VNDetectFaceRectanglesRequest() let faceDetectionRequestHandler = VNSequenceRequestHandler() Here, we are simply creating the request and handler; as discussed in the preceding code, the request encapsulates the type of image analysis while the handler is responsible for executing the request. Next, we will get faceDetectionRequestHandler to run faceDetectionRequest; add the following code: try? faceDetectionRequestHandler.perform( [faceDetectionRequest], on: images[faceIdx].cgImage!, orientation: CGImagePropertyOrientation(images[faceIdx].imageOrientation)) The perform function of the handler can throw an error if it fails; for this reason, we wrap the call with try? at the beginning of the statement and can interrogate the error property of the handler to identify the reason for failing. We pass the handler a list of requests (in this case, only our faceDetectionRequest), the image we want to perform the analysis on, and, finally, the orientation of the image that can be used by the request during analysis. Once the analysis is done, we can inspect the observation obtained through the results property of the request itself, as shown in the following code: if let faceDetectionResults = faceDetectionRequest.results as? [VNFaceObservation]{ for face in faceDetectionResults{ // ADD THE NEXT SNIPPET OF CODE HERE } } The type of observation is dependent on the analysis; in this case, we're expecting a VNFaceObservation. Hence, we cast it to the appropriate type and then iterate through all the observations. Next, we will take each recognized face and extract the bounding box. Then, we'll proceed to draw it in the image (using an extension method of UIImageView found within the UIImageViewExtension.swift file). Add the following block within the for loop shown in the preceding code: if let currentImage = imageView.image{ let bbox = face.boundingBox let imageSize = CGSize( width:currentImage.size.width, height: currentImage.size.height) let w = bbox.width * imageSize.width let h = bbox.height * imageSize.height let x = bbox.origin.x * imageSize.width let y = bbox.origin.y * imageSize.height let faceRect = CGRect( x: x, y: y, width: w, height: h) let invertedY = imageSize.height - (faceRect.origin.y + faceRect.height) let invertedFaceRect = CGRect( x: x, y: invertedY, width: w, height: h) imageView.drawRect(rect: invertedFaceRect) } We can obtain the bounding box of each face via the let boundingBox property; the result is normalized, so we then need to scale this based on the dimensions of the image. For example, you can obtain the width by multiplying boundingBox with the width of the image: bbox.width * imageSize.width. Next, we invert the y axis as the coordinate system of Quartz 2D is inverted with respect to that of UIKit's coordinate system, as shown in this diagram: We invert our coordinates by subtracting the bounding box's origin and height from height of the image and then passing this to our UIImageView to render the rectangle. Click on the eye icon in the right-hand panel in line with the statement imageView.drawRect(rect: invertedFaceRect) to preview the results; if successful, you should see something like the following: An alternative to inverting the face rectangle would be to use an AfflineTransform, such as: var transform = CGAffineTransform(scaleX: 1, y: -1) transform = transform.translatedBy(x: 0, y: -imageSize.height) let invertedFaceRect = faceRect.apply(transform) This approach leads to less code and therefore less chances of errors. So, it is the recommended approach. The long approach was taken previously to help illuminate the details. As a designer and builder of intelligent systems, it is your task to interpret these results and present them to the user. Some questions you'll want to ask yourself are as follows: What is an acceptable threshold of a probability before setting the class as true? Can this threshold be dependent on probabilities of other classes to remove ambiguity? That is, if Sad and Happy have a probability of 0.3, you can infer that the prediction is inaccurate, or at least not useful. Is there a way to accept multiple probabilities? Is it useful to expose the threshold to the user and have it manually set and/or tune it? These are only a few questions you should ask. The specific questions and their answers will depend on your use case and users. At this point, we have everything we need to preprocess and perform inference We briefly explored some use cases showing how emotion recognition could be applied. For a detailed overview of this experiment, check out our book, Machine Learning with Core ML to further implement Core ML for visual-based applications using the principles of transfer learning and neural networks. Amazon Rekognition can now ‘recognize’ faces in a crowd at real-time 5 cool ways Transfer Learning is being used today My friend, the robot: Artificial Intelligence needs Emotional Intelligence
Read more
  • 0
  • 0
  • 8359

article-image-erasure-coding-cold-storage
Packt
07 Jun 2017
20 min read
Save for later

Erasure coding for cold storage

Packt
07 Jun 2017
20 min read
In this article by Nick Frisk, author of the book Mastering Ceph, we will get acquainted with erasure coding. Ceph's default replication level provides excellent protection against data loss by storing three copies of your data on different OSD's. The chance of losing all three disks that contain the same objects within the period that it takes Ceph to rebuild from a failed disk, is verging on the extreme edge of probability. However, storing 3 copies of data vastly increases both the purchase cost of the hardware but also associated operational costs such as power and cooling. Furthermore, storing copies also means that for every client write, the backend storage must write three times the amount of data. In some scenarios, either of these drawbacks may mean that Ceph is not a viable option. Erasure codes are designed to offer a solution. Much like how RAID 5 and 6 offer increased usable storage capacity over RAID1, erasure coding allows Ceph to provide more usable storage from the same raw capacity. However also like the parity based RAID levels, erasure coding brings its own set of disadvantages. (For more resources related to this topic, see here.) In this article you will learn: What is erasure coding and how does it work Details around Ceph's implementation of erasure coding How to create and tune an erasure coded RADOS pool A look into the future features of erasure coding with Ceph Kraken release What is erasure coding Erasure coding allows Ceph to achieve either greater usable storage capacity or increase resilience to disk failure for the same number of disks versus the standard replica method. Erasure coding achieves this by splitting up the object into a number of parts and then also calculating a type of Cyclic Redundancy Check, the Erasure code, and then storing the results in one or more extra parts. Each part is then stored on a separate OSD. These parts are referred to as k and m chunks, where k refers to the number of data shards and m refers to the number of erasure code shards. As in RAID, these can often be expressed in the form k+m or 4+2 for example. In the event of an OSD failure which contains an objects shard which isone of the calculated erasure codes, data is read from the remaining OSD's that store data with no impact. However, in the event of an OSD failure which contains the data shards of an object, Ceph can use the erasure codes to mathematically recreate the data from a combination of the remaining data and erasure code shards. k+m The more erasure code shards you have, the more OSD failure's you can tolerate and still successfully read data. Likewise the ratio of k to m shards each object is split into, has a direct effect on the percentage of raw storage that is required for each object. A 3+1 configuration will give you 75% usable capacity, but only allows for a single OSD failure and so would not be recommended. In comparison a three way replica pool, only gives you 33% usable capacity. 4+2 configurations would give you 66% usable capacity and allows for 2 OSD failures. This is probably a good configuration for most people to use. At the other end of the scale a 18+2 would give you 90% usable capacity and still allows for 2 OSD failures. On the surface this sounds like an ideal option, but the greater total number of shards comes at a cost. The higher the number of total shards has a negative impact on performance and also an increased CPU demand. The same 4MB object that would be stored as a whole single object in a replicated pool, is now split into 20 x 200KB chunks, which have to be tracked and written to 20 different OSD's. Spinning disks will exhibit faster bandwidth, measured in MB/s with larger IO sizes, but bandwidth drastically tails off at smaller IO sizes. These smaller shards will generate a large amount of small IO and cause additional load on some clusters. Also its important not to forget that these shards need to be spread across different hosts according to the CRUSH map rules, no shard belonging to the same object can be stored on the same host as another shard from the same object. Some clusters may not have a sufficient number hosts to satisfy this requirement. Reading back from these high chunk pools is also a problem. Unlike in a replica pool where Ceph can read just the requested data from any offset in an object, in an Erasure pool, all shards from all OSD's have to be read before the read request can be satisfied. In the 18+2 example this can massively amplify the amount of required disk read ops and average latency will increase as a result. This behavior is a side effect which tends to only cause a performance impact with pools that use large number of shards. A 4+2 configuration in some instances will get a performance gain compared to a replica pool, from the result of splitting an object into shards.As the data is effectively striped over a number of OSD's, each OSD is having to write less data and there is no secondary and tertiary replica's to write. How does erasure coding work in Ceph As with Replication, Ceph has a concept of a primary OSD, which also exists when using erasure coded pools. The primary OSD has the responsibility of communicating with the client, calculating the erasure shards and sending them out to the remaining OSD's in the Placement Group (PG) set. This is illustrated in the diagram below: If an OSD in the set is down, the primary OSD, can use the remaining data and erasure shards to reconstruct the data, before sending it back to the client. During read operations the primary OSD requests all OSD's in the PG set to send their shards. The primary OSD uses data from the data shards to construct the requested data, the erasure shards are discarded. There is a fast read option that can be enabled on erasure pools, which allows the primary OSD to reconstruct the data from erasure shards if they return quicker than data shards. This can help to lower average latency at the cost of slightly higher CPU usage. The diagram below shows how Ceph reads from an erasure coded pool: The next diagram shows how Ceph reads from an erasure pool, when one of the data shards is unavailable. Data is reconstructed by reversing the erasure algorithm using the remaining data and erasure shards. Algorithms and profiles There are a number of different Erasure plugins you can use to create your erasure coded pool. Jerasure The default erasure plugin in Ceph is the Jerasure plugin, which is a highly optimized open source erasure coding library. The library has a number of different techniques that can be used to calculate the erasure codes. The default is Reed Solomon and provides good performance on modern processors which can accelerate the instructions that the technique uses. Cauchy is another technique in the library, it is a good alternative to Reed Solomon and tends to perform slightly better. As always benchmarks should be conducted before storing any production data on an erasure coded pool to identify which technique best suits your workload. There are also a number of other techniques that can be used, which all have a fixed number of m shards. If you are intending on only having 2 m shards, then they can be a good candidate, as there fixed size means that optimization's are possible lending to increased performance. In general the jerasure profile should be prefer in most cases unless another profile has a major advantage, as it offers well balanced performance and is well tested. ISA The ISA library is designed to work with Intel processors and offers enhanced performance. It too supports both Reed Solomon and Cauchy techniques. LRC One of the disadvantages of using erasure coding in a distributed storage system is that recovery can be very intensive on networking between hosts. As each shard is stored on a separate host, recovery operations require multiple hosts to participate in the process. When the crush topology spans multiple racks, this can put pressure on the inter rack networking links. The LRC erasure plugin, which stands for Local Recovery Codes, adds an additional parity shard which is local to each OSD node. This allows recovery operations to remain local to the node where a OSD has failed and remove the need for nodes to receive data from all other remaining shard holding nodes. However the addition of these local recovery codes does impact the amount of usable storage for a given number of disks. In the event of multiple disk failures, the LRC plugin has to resort to using global recovery as would happen with the jerasure plugin. SHingled Erasure Coding The SHingled Erasure Coding (SHEC) profile is designed with similar goals to the LRC plugin, in that it reduces the networking requirements during recovery. However instead of creating extra parity shards on each node, SHEC shingles the shards across OSD's in an overlapping fashion. The shingle part of the plugin name represents the way the data distribution resembles shingled tiles on a roof of a house. By overlapping the parity shards across OSD's, the SHEC plugin reduces recovery resource requirements for both single and multiple disk failures. Where can I use erasure coding Since the Firefly release of Ceph in 2014, there has been the ability to create a RADOS pool using erasure coding. There is one major thing that you should be aware of, the erasure coding support in RADOS does not allow an object to be partially updated. You can write to an object in an erasure pool, read it back and even overwrite it whole, but you cannot update a partial section of it. This means that erasure coded pools can't be used for RBD and CephFS workloads and is limited to providing pure object storage either via the Rados Gateway or applications written to use librados. The solution at the time was to use the cache tiering ability which was released around the same time, to act as a layer above an erasure coded pools that RBD could be used. In theory this was a great idea, in practice, performance was extremely poor. Every time an object was required to be written to, the whole object first had to be promoted into the cache tier. This act of promotion probably also meant that another object somewhere in the cache pool was evicted. Finally the object now in the cache tier could be written to. This whole process of constantly reading and writing data between the two pools meant that performance was unacceptable unless a very high percentage of the data was idle. During the development cycle of the Kraken release, an initial implementation for support for direct overwrites on n erasure coded pool was introduced. As of the final Kraken release, support is marked as experimental and is expected to be marked as stable in the following release. Testing of this feature will be covered later in this article. Creating an erasure coded pool Let's bring our test cluster up again and switch into SU mode in Linux so we don't have to keep prepending sudo to the front of our commands Erasure coded pools are controlled by the use of erasure profiles, these control how many shards each object is broken up into including the split between data and erasure shards. The profiles also include configuration to determine what erasure code plugin is used to calculate the hashes. The following plugins are available to use <list of plugins> To see a list of the erasure profiles run # cephosd erasure-code-profile ls You can see there is a default profile in a fresh installation of Ceph. Lets see what configuration options it contains # cephosd erasure-code-profile get default The default specifies that it will use the jerasure plugin with the Reed Solomon error correcting codes and will split objects into 2 data shards and 1 erasure shard. This is almost perfect for our test cluster, however for the purpose of this exercise we will create a new profile. # cephosd erasure-code-profile set example_profile k=2 m=1 plugin=jerasure technique=reed_sol_van # cephosd erasure-code-profile ls You can see our new example_profile has been created. Now lets create our erasure coded pool with this profile: # cephosd pool create ecpool 128 128 erasure example_profile The above command instructs Ceph to create a new pool called ecpool with a 128 PG's. It should be an erasure coded pool and should use our "example_profile" we previously created. Lets create an object with a small text string inside it and the prove the data has been stored by reading it back: # echo "I am test data for a test object" | rados --pool ecpool put Test1 – # rados --pool ecpool get Test1 - That proves that the erasure coded pool is working, but it's hardly the most exciting of discoveries. Lets have a look to see if we can see what's happening at a lower level. First, find out what PG is holding the object we just created # cephosd map ecpoolTest1 The result of the above command tells us that the object is stored in PG 3.40 on OSD's1, 2 and 0. In this example Ceph cluster that's pretty obvious as we only have 3 OSD's, but in larger clusters that is a very useful piece of information. We can now look at the folder structure of the OSD's and see how the object has been split. The PG's will likely be different on your test cluster, so make sure the PG folder structure matches the output of the "cephosd map" command above. ls -l /var/lib/ceph/osd/ceph-2/current/1.40s0_head/ # ls -l /var/lib/ceph/osd/ceph-1/current/1.40s1_head/ # ls -l /var/lib/ceph/osd/ceph-0/current/1.40s2_head/                 total 4 Notice how the PG directory names have been appended with the shard number, replicated pools just have the PG number as their directory name. If you examine the contents of the object files, you will see our text string that we entered into the object when we created it. However due to the small size of the text string, Ceph has padded out the 2nd shard with null characters and the erasure shard hence will contain the same as the first. You can repeat this example with a new object containing larger amounts of text to see how Ceph splits the text into the shards and calculates the erasure code. Overwrites on erasure code pools with Kraken Introduced for the first time in the Kraken release of Cephas an experimental feature, was the ability to allow partial overwrites on erasure coded pools. Partial overwrite support allows RBD volumes to be created on erasure coded pools, making better use of raw capacity of the Ceph cluster. In parity RAID, where a write request doesn't span the entire stripe, a read modify write operation is required. This is needed as the modified data chunks will mean the parity chunk is now incorrect. The RAID controller has to read all the current chunks in the stripe, modify them in memory, calculate the new parity chunk and finally write this back out to the disk. Ceph is also required to perform this read modify write operation, however the distributed model of Ceph increases the complexity of this operation.When the primary OSD for a PG receives a write request that will partially overwrite an existing object, it first works out which shards will be not be fully modified by the request and contacts the relevant OSD's to request a copy of these shards. The primary OSD then combines these received shards with the new data and calculates the erasure shards. Finally the modified shards are sent out to the respective OSD's to be committed. This entire operation needs to conform the other consistency requirements Ceph enforces, this entails the use of temporary objects on the OSD, should a condition arise that Ceph needs to roll back a write operation. This partial overwrite operation, as can be expected, has a performance impact. In general the smaller the write IO's, the greater the apparent impact. The performance impact is a result of the IO path now being longer, requiring more disk IO's and extra network hops. However, it should be noted that due to the striping effect of erasure coded pools, in the scenario where full stripe writes occur, performance will normally exceed that of a replication based pool. This is simply down to there being less write amplification due to the effect of striping. If performance of an Erasure pool is not suitable, consider placing it behind a cache tier made up of a replicated pool. Despite partial overwrite support coming to erasure coded pools in Ceph, not every operation is supported. In order to store RBD data on an erasure coded pool, a replicated pool is still required to hold key metadata about the RBD. This configuration is enabled by using the –data-pool option with the rbd utility. Partial overwrite is also not recommended to be used with Filestore. Filestore lacks several features that partial overwrites on erasure coded pools uses, without these features extremely poor performance is experienced. Demonstration This feature requires the Kraken release or newer of Ceph. If you have deployed your test cluster with the Ansible and the configuration provided, you will be running Ceph Jewel release. The following steps show how to use Ansible to perform a rolling upgrade of your cluster to the Kraken release. We will also enable options to enable experimental options such as bluestore and support for partial overwrites on erasure coded pools. Edit your group_vars/ceph variable file and change the release version from Jewel to Kraken. Also add: ceph_conf_overrides: global: enable_experimental_unrecoverable_data_corrupting_features: "debug_white_box_testing_ec_overwrites bluestore" And to correct a small bug when using Ansible to deploy Ceph Kraken, add: debian_ceph_packages: - ceph - ceph-common - ceph-fuse To the bottom of the file run the following Ansible playbook: ansible-playbook -K infrastructure-playbooks/rolling_update.yml Ansible will prompt you to make sure that you want to carry out the upgrade, once you confirm by entering yes the upgrade process will begin. Once Ansible has finished, all the stages should be successful as shown below: Your cluster has now been upgraded to Kraken and can be confirmed by running ceph -v on one of yours VM's running Ceph. As a result of enabling the experimental options in the configuration file, every time you now run a Ceph command, you will be presented with the following warning. This is designed as a safety warning to stop you running these options in a live environment, as they may cause irreversible data loss. As we are doing this on a test cluster, that is fine to ignore, but should be a stark warning not to run this anywhere near live data. The next command that is required to be run is to enable the experimental flag which allows partial overwrites on erasure coded pools. DO NOT RUN THIS ON PRODUCTION CLUSTERS cephosd pool get ecpooldebug_white_box_testing_ec_overwrites true Double check you still have your erasure pool called ecpool and the default RBD pool # cephosdlspools 0 rbd,1ecpool, And now create the rbd. Notice that the actual RBD header object still has to live on a replica pool, but by providing an additional parameter we can tell Ceph to store data for this RBD on an erasure coded pool. rbd create Test_On_EC --data-pool=ecpool --size=1G The command should return without error and you now have an erasure coded backed RBD image. You should now be able to use this image with any librbd application. Note: Partial overwrites on Erasure pools require Bluestore to operate efficiently. Whilst Filestore will work, performance will be extremely poor. Troubleshooting the 2147483647 error An example of this error is shown below when running the ceph health detail command. If you see 2147483647 listed as one of the OSD's for an erasure coded pool, this normally means that CRUSH was unable to find a sufficient number of OSD's to complete the PG peering process. This is normally due to the number of k+m shards being larger than the number of hosts in the CRUSH topology. However, in some cases this error can still occur even when the number of hosts is equal or greater to the number of shards. In this scenario it's important to understand how CRUSH picks OSD's as candidates for data placement. When CRUSH is used to find a candidate OSD for a PG, it applies the crushmap to find an appropriate location in the crush topology. If the result comes back as the same as a previous selected OSD, Ceph will retry to generate another mapping by passing slightly different values into the crush algorithm. In some cases if there is a similar number of hosts to the number of erasure shards, CRUSH may run out of attempts before it can suitably find correct OSD mappings for all the shards. Newer versions of Ceph has mostly fixed these problems by increasing the CRUSH tunable choose_total_tries. Reproducing the problem In order to aid understanding of the problem in more detail, the following steps will demonstrate how to create an erasure coded profile that will require more shards than our 3 node cluster can support. Firstly, like earlier in the articlecreate a new erasure profile, but modify the k/m parameters to be k=3 m=1: $ cephosd erasure-code-profile set broken_profile k=3 m=1 plugin=jerasure technique=reed_sol_van And now create a pool with it: $ cephosd pool create broken_ecpool 128 128 erasure broken_profile If we look at the output from ceph -s, we will see that the PG's for this new pool are stuck in the creating state. The output of ceph health detail, shows the reason why and we see the 2147483647 error. If you encounter this error and it is a result of your erasure profile being larger than your number of hosts or racks, depending on how you have designed your crushmap. Then the only real solution is to either drop the number of shards, or increase number of hosts. Summary In this article you have learnt what erasure coding is and how it is implemented in Ceph. You should also have an understanding of the different configuration options possible when creating erasure coded pools and their suitability for different types of scenarios and workloads. Resources for Article: Further resources on this subject: Ceph Instant Deployment [article] Working with Ceph Block Device [article] GNU Octave: Data Analysis Examples [article]
Read more
  • 0
  • 0
  • 8358