The housing crisis deeply impacted millions of Americans, and today the effects are still being felt. For the many problems facing communities, from wage disparity to affordable housing, there isn't a single fits-all solution.
That's why we are very excited about the work that Woodstock Foundation is doing to support fair housing policies. Today they launch a nuanced look at housing and income disparity in Illinois in the form of a new map-based open data website.
The site brings together 74 datasets on the well-being of local communities. It is a good roadmap for anyone working in housing justice in Illinois. Community organizations can explore the average amount of mortgage debt people take on and the rate of foreclosure filings in the Chicago six-county region and elsewhere in Illinois to inform their decisions on where to focus their work. Some data sets go back to 2008.
Where the highest income census tracts are in the Chicago six-county region
Serving complex data through static JSON
Housing data is complex, and Woodstock has gathered some amazingly granular statistics about housing data in Illinois. Splitting this data into a format that we could serve over the web proved a difficult challenge. Woodstock is also a small nonprofit, and we wanted to ease as much as possible the burden of maintaining a complicated website.
So we wrote a python library to break up their spreadsheets into JSON. Every time you switch to a new facet of housing data, every time you view a different year or category of that data, your browser incrementally downloads a new JSON file. Although those files in aggregate would take ages to load, individually they are manageable. Those scripts, along with the rest of the site, are open-source.
To further reduce load times, we use the topojson spec to reduce the size of geographic boundary data. This allows us to separate geographic data from numerical data, so you only download those complex census tract boundaries once. The code that runs in your browser than re-connects those boundaries to the housing or mortgage data you select on the fly.
Using vector-based geographical boundaries has other benefits. It allowed us to use a mouse click on an overlaying geographic boundary and a point-in-polygon test to find, for example, which congressional representative is responsible for which census tract.
Detecting congressional boundaries
Census tracts over multiple years
Census tracts change a lot over census years, and this can be a problem when mapping a multi-year data set that covers more than a single census geography file. Attempting to compare the data between those years that use different tract definitions can be tricky. Tract ID, or FIPS codes can either refer to a different neighborhood or disappear entirely.
The Census releases relationship files that show where these changes and additions occur. Using this, we created a tool that overlays 2000 and 2010 census tracts, and shows differences between the two years.
Currently the project covers only Illinois, but the code is available on GitHub and ready for you to fork and contribute your own state.
Recently, I was faced with an interesting challenge; develop a system for importing thousands of hand-build sites into Drupal. One of the tools that we encountered in our research was import.io, a web data platform and web scraping tool. This blog post will take you through the steps of using import.io to bring content into a Drupal site.
Find out how to change the background colour of your website with BlueGriffon, the open source (and free) web editor, in this chapter of the BlueGriffon tutorial series.
Toxic algal bloom in Lake Erie
To address climate change and promote environmental justice, we need better tools to understand our changing planet. That's why we are delighted to help host the DC EcoHack with WRI at the Mapbox Garage.
EcoHack is an event to bring together a diverse community of scientists, hackers, designers and others who want to tell stories and create tools to protect our environment. The event is open to people of all skill levels. As long you're interested in using technology to improve and better understand our natural environment, we'd love to see you there.
If you're in DC, register and come join us at the MapBox Garage in DC. Our friends at Mapbox are also hosting the San Francisco EcoHack and EcoHacks will also take place in Sydney, Cambridge, New York, and Madrid.
Drew and Marc will be using some of the time to work on landsat-util, an open source tool that makes it easier to work with open satellite imagery. We are also keen to help on projects using green energy investment data and tracking natural disasters.
Hope to see you there!
For ToadCast 28 we have special guest Chris Bloom!
Chris teaches our Director of Development Dan Linn, about new Front-End technologies.
Now you can see the bonus features of the podcast on our Youtube page!
Chris Bloom gives video tutorial on the technologies mentioned below!
- Souls of Mischief
- Run the Jewels 2
- Serge Severe
- Pretty Lights
- Glitch Mob
- Jellyfish Brigade
ToadCast can be found on iTunes. Check us out and give us a review/rating.
Satellite imagery over Monrovia, an area that has seen a significant burden of Ebola cases.
Today, Marc and I are heading up to New York to attend the International Conference of Crisis Mappers. Open crisis mapping is growing up. We are seeing greater demand for maps and data for crisis response and preparation. ICCM 2014 will be an important place to discuss how we can grow to handle this need, how we better generate real collaboration from data, and how we build infrastructure that is usable and inclusive.
We're looking forward to talking about OpenStreetMap, satellites, and open data; topics that are critical in the midst of the response to the Ebola outbreak in West Africa. Reach out to @nas_smith or @kamicut on Twitter if you want to chat.
Drupal 8 is the first time we introduced the concept of formal initiatives and initiative leads. Over the course of these Drupal 8 initiatives we learned a lot and people are floating several ideas to increase the initiatives' success and provide Drupal initiative leads with more support. As we grow, it is crucial that we evolve our tools, our processes, and our organizational design based on these learnings. We've done so in the past and we'll continue to do so going forward.
But let's be honest, no matter how much support we provide, leading a Drupal initiative will unquestionably remain difficult and overwhelming. As a Drupal initiative lead, you are asked to push forward some of the most difficult and important parts of Drupal.
You will only succeed if you are able to build a strong team of volunteers that is willing to be led by you. You have to learn how to inspire and motivate by articulating a vision. You establish credibility by setting clear objectives and roadmaps in partnership with others. You have to motivate, guide and empower people to participate. You have to plan and over-communicate.
Not only do you have to worry about building and leading a team, you also have to make sure the rest of the community has shared goals and that everyone impacted has a shared understanding of why those decisions are being made. You use data, ideas and feedback from different sources to inform and convince people of your direction. Your "soft skills" are more important than your "hard skills". Regardless, you will lose many battles. You only "win" when you remain open to feedback and value change and collaboration. To lead a community, you need both a thick skin and a big heart.
Success is never a coincidence. You put in long hours to try and keep your initiative on track. You need relentless focus on doing whatever is necessary to succeed; to be the person who fills all the gaps and helps others to be successful. Instead of just doing the things you love doing most, you find yourself doing mundane tasks like updating spreadsheets or planning a code sprint to help others be successful. In fact, you might need to raise money for your code sprint. And if you succeed, you still don't have enough money to achieve what is possible and you feel the need to raise even more. You'll be brushing aside or knocking down obstacles in your path, and taking on jobs and responsibilities you have never experienced before.
Your objectives will constantly shift as Drupal itself iterates and evolves. You will want to go faster and you will struggle with the community processes. Imagine working on something for a month and then having to throw it out completely because you realize it doesn't pass. Frustration levels will be off the charts. Your overall goal of achieving the perfect implementation might never be achieved and that feeling haunts you for weeks or months. You will feel the need to vent publicly, and you probably will. At the worst moments, you'll think about stepping down. In better times, you realize that if most of your initiative succeeds it could take years of follow-up work. You will learn a lot about yourself; you learn that you are bad at many things and really good at other things.
Leading is incredibly hard and yet, it will be one of the best thing you ever did. You work with some of the finest, brightest, and most passionate people in the world. You will see tangible results of your hard work and you will impact and help hundreds of thousands of people for the next decade. There is no better feeling than when you inspire or when you help others succeed. Leading is hard, but many of you will look back at your time and say this was the most gratifying thing you ever did. You will be incredibly proud of yourself, and the community will be incredibly proud of you. You will become a better leader, and that will serve you for the rest of your life.
Working embed. US voters, find your polling location
In the US, and around the world, it can be confusing figuring out where you should vote and which races you are eligible to vote on. In the US, going to the wrong polling station is a hassle. This hassle can be prohibitive, particularly if you are disabled or rely on public transportation. In countries coming out of conflict, going to the polling station can be a brave and risky act. You'd better be at the right place when you get there.
Open data is helping to get voters the information they need to participate. By opening up data about voting locations and process, States can involve private actors, such as The Pew Trusts and Google, as partners providing accurate information about where to vote. The VIP (Voter Information Project) is an embeddable tool, based on an open source stack, that relies on an open API. This allows other groups to repackage and distribute this information to their audiences.
By opening up the data, States are no longer solely responsible for getting voters to the right polling place. Voters should be tripping over this information in every Google search, Foursquare check-in, community message board, and favorite blogs.
"Usability and Voting" by ericgundersen
At Development Seed, design is not about pushing pixels or passing a perfect mockup to the next person. It is about truly understanding — and sometimes defining — a problem, working out a systemic solution with visual and interactive components. You will be brainstorming solutions with our strategists and turning them into sketches, websites, and data visualizations with fellow developers.
We build new ways to help people make decisions — impacting policies and creating transparency on all levels. We are hiring a designer who is a doer and a thinker, eager to join our mission.
- an artist; you have a favorite medium to express your ideas, be it ink, paint, vector or gif
- excited about the web, particularly how the representation of information on screens can inform people’s decisions
- eager to work with data and the patterns it leads to
- curious and hungry to learn new subjects and skills
You know how to:
- ask good questions and get to the heart of a problem
- illustrate abstract concepts and workflows in visual forms
- use the right font and color at the right time, knowing that aesthetics is derived from your communication goal, the information, and the medium
Experiences with any of the following will be a plus:
- Web maps (such as a map made with MapBox Studio)
- Responsive web frameworks
- D3.js for visualization (bar chart counts)
- Static site generation (Jekyll, Flask, etc)
But don't let any of that scare you. If your design chops are good, we will work with you to tech up on everything you need to know.
Please send your portfolio site, and links to three projects, to firstname.lastname@example.org with “Designer” in your subject.
Now that we have a fairly good idea how Drupal 8 and data looks let's discuss what can MongoDB provide and why would you want to run it. In Drupal 8, every kind of data can be stored independently. I fully expect that people will indeed mix storages. For example, D8 by default runs a config query on every page to find the blocks to be displayed for the current theme. Again, by default, config entities are stored as serialized PHP arrays so the only way a query like that can run is to load every single block entity from the database and iterate over them. This can get quite slow with hundreds of blocks -- don't forget that a block entity just stores the placement of a block and the same block can be placed several times. Also, after finding the blocks for the current theme, visibility rules (path, node types, roles) are applied again in PHP. If the configuration is stored in MongoDB then all this can become an indexed, practically instant query. Configuration storage, after all, is just storing and querying and retrieving arrays and MongoDB is really, really good at that. Because of this, I expect many sites to pick MongoDB for their configuration storage. I also expect that because of the simplicity of cross-entity type JOINs, many people will stick with SQL for their content storage. Although it must be noted that the choice can be made per entity type and as MongoDB stores complete entities it is able to index even those queries that the SQL storage can not.
There are some simpler storages which are also a good fit for MongoDB: sessions because of the write performance and logging because MongoDB has a capped collection (a circular buffer) feature so you will always have the latest N messages and never too much.
In this episode Kyle is joined by a few members of Lullabot's front-end army.
Dauria Geo is just completing new design specs for their Perseus satellite constellations. Perseus-O, a constellation of 8 satellites, will provide daily global coverage of all arable land at 22 meters resolution (meaning each pixel represents 22 meters on the ground). With the same spectral bands as Landsat this new imagery will be able to measure crop health and flooding. The Perseus-HD constellation of 20 satellites will provide daily images of all urban and arable land at 2.5 meters resolution -- showing roads, buildings, ships, and fields.
In addition to having their own satellites, Dauria Geo will make Landsat and MODIS open data sources accessible through their API. They have established partnerships with industry leaders like Deimos in Spain, EIAST in Dubai, and Eye Innovation in China to provide a variety of resolution, coverage, sensor, and freshness of imagery, offering a unique balance between resolution and timely revisit. This is super exciting for our team as we work to expand where we source imagery for NGOs to process in their pipelines.
The new technical specs mean Dauria Geo moves into the next build phase, and is on target to begin launching the Perseus constellation in 2015. We're collaborating with Dauria Geo now as they build out integration and visualization tools -- from antenna to API, that makes image acquisition, analysis, delivery, and integration easier for both NGOs and enterprise. Dauria Geo is building an API to empower developers to access fresh and historical imagery, compute needed data layers on the cloud, and harvest data in ready-to-use format. Their cloud platform can do heavy analysis and feed data directly into applications. By directly integrating with the Mapbox API, we can quickly deploy sophisticated and beautiful applications from agriculture to disaster response using the platforms that developers are already building on.
We'd love to see more satellite providers compete on ease of integrating their data. We'll be helping Dauria Geo to review their API to make it developer friendly and we will build open source tools on top Dauria's API. These tools will serve as open templates for integrating Dauria with tools like Mapbox to quickly build powerful, data-rich sites. This is really positive move for the industry and for users.
In our last post we used CasperJS to rapidly test the user interface of a website. Now we will build on these skills and add a familiar element into the mix: Drupal. Like any framework, Drupal offers many predictable, standard behaviors which we can take advantage of. Using this predictability, we can easily test many behaviors including logged-in activity such as posting content.
Seeing Long Term Technology Adoption as Evolution
October 29th, 2014
Photo Credit: http://www.bikesexual.org/cm/pic/foto/dino/dino_on_bike.jpg
Much like an evolutionary tree our goal in technology adoption is to continue to move forward and evolve, rather than getting caught in a dead end. In the natural world, becoming bigger can be good but can lead to extinction events should the environment or food source change. Right now we are in a technology Jurassic period; oxygen and plants are plentiful and everyone can have wildly different ways to succeed. That said, let's examine the barriers to success as well as competitive advantages and the criteria we should use as predictors.
- Success Rate
If a platform or language are complex (Drupal & Ruby are possible examples here) adoption will be slow. The smarter you are, the less likely you are to see this issue, as complexity is relative. However on the whole this hurts because it reduces the pool of available workers and generally pushes up rates and cost.
The more documentation that is publicly available, the more chance of overcoming any issues with Complexity, helping to facilitate growth in the worker pool. This is generally a sign of a highly active Community, because it involves not just engineers but people who generally play support roles. This is an interesting lever, as it's relatively easy to influence (write more documentation) and yet it's value ages rapidly as the technology evolves.
As a technology becomes more successful, the available pool of workers is reduced and wages increase. This is inevitable, as learning of new technology takes months or years, while the demand can grow as fast as word of mouth can recommend it. Initially in this phase, the market can and will absorb the increased costs - employers and clients will pay more - but eventually cost increase will cause the demand itself to wane, but this occurs gradually, first as a reduction of the number of businesses willing (and able) to pay the prevailing rate, and then eventually starvation. This is a relative measure based on the environments that can be accessed. Looking at Drupal as an example, there is plenty more being paid by Oracle customers for less value, but if Drupal vendors can't reach Oracle markets and continue to need to compete with the less expensive options (like WordPress) they face stiff competition and will eventually be outcompeted. I believe targeted education, such as vocational programming, is a way to artificially keep costs down, though this needs to be fixed on a market-level rather then just for an individual employer.
Scarcity in the workforce, and the corresponding increase in costs is the precursor of scarcity for the technology as a solution that is available to solve problems. If there aren't enough people to build websites in Drupal, they will be built on other platforms, resulting is larger utilization for the most available platforms. This availability/scarcity of workers for a platform is amplified over time, because any project is a group effort. Every time a platform is used to solve a business problem more people are exposed to the technology and build it into their consciousness as a possible solution for their next project.
5. Success Rate
Every time a technology succeeds on a project regardless of the reason, that technology falls into the "win" category for everyone to work on the project. Every time a project fails, the technology is invariably linked to failure and is less likely to be accepted as a viable solution for the next project. This means success rate on a market-level (not just individual companies) combined with scarcity/abundance is a key factor in determining future total usage and community.
There are a number of measures of community including:
- total usage (how many sites/developers) - votes for your software
- code contributions - the engine of forward progress
- organizational infrastructure - likelihood the evolution path will be a smart one
- community events - engagement
Looking at this from an evolutionary stand point, total usage is the most important stat, as "winning" should be defined as the being the most prolific platform. Being the "best" is subjective, and evolutionarily speaking is less important than being "more". Code contributions should be equated to how much evolutionary potential there is, which is an important factor, but only represents future potential. Organizational structure is less important, though I would liken it to intelligence - how well can we use the tools we have at our disposal? Community events are an indicator of engagement and show potential for increasing usage and code contributions.
Ultimately technologies that want to survive should pay close attention to these levers and predictors. Programmers, technologists and companies that want to be on a winning branch of the evolutionary tree should as well. History and nature have not been kind to those who fail to adapt.
Marc Farra has joined the Development Seed team. Marc loves to experiment with image processing, arduino sensors, and data infrastructure. He is going to help us explore new ways to collect and process data.
I first ran into Marc in Beirut. At the time he was running Lamba Labs, a hacker space in Beirut that was sowing the seeds of Maker culture and open data advocacy in Lebanon. A year later, he took the Afghanistan polling station locations we posted on Github and started to build a mobile app for Afghans to locate their nearest polling station.
Reversing climate change means investing in green energy, and as the sustainable sector grows, ensuring it grows in both developing and developed countries. Today the Fondo Multilateral de Inversiones and Bloomberg New Energy Finance are launching a vastly expanded Climatescope, to provide open data about green energy investment in 55 countries. The data provided by Climatescope creates an information-rich environment for green energy investors. It also provides valuable data on clean energy policies for activists and policymakers.
Opening Climate Investment Data
We worked on the Climatescope website with Flipside, a smart, new open source technology shop based in Lisbon.
The site takes a very thoughtful approach to opening information. All the data powering the Climatescope site is available through an open API, which you can easily integrate into your own applications. The full dataset is also available for analysis. On almost every page lives a download button that provides a CSV file containing whatever you happen to be viewing.
Most importantly, FOMIN got the licensing right. The data is licensed CC-BY. It can be used (with attribution) by anyone, even for commercial purposes. This is critical when you want data to encourage commercial activity. Moreover the website itself is also open and is licensed GPL 3.0. The entire site can be forked by other open source projects.
Dynamic Static Websites
Like many of the sites we (and Flipside) build these days, Climatescope is a fully interactive site without a database or a heavy CMS. Climatescope users can manipulate, interrogate, and download the data on any device and in low bandwidth requirements. The site uses Jekyll, Angular, and D3 (among other tools) and is hosted on Github. Read more on our approach to CMS-free websites.
People have different priorities when evaluating the environment for clean energy. The site is designed for a range of users, from activists to journalists, politicians, environmentalists, and the curious. FOMIN is committed to giving Climatescope users full control over how much weight each metric carries. To accomodate this, we built simple, intuitive sliders. Movement in one slider spreads the difference evenly across the other three factors. You can lock any slider to make it easier to hit an exact breakdown.
Hacking for the Planet
Have some data or coding skills? Care about the planet? Consider joining an EcoHack near you on Nov 15-16. We are hosting the DC EcoHack with WRI. EcoHacks are also happening in Sydney, Cambridge, New York, Madrid, and San Francisco.
Adding a contact form to a website can be a formidable task for many newcomers. This article shows you how you can do it using Dreamweaver CS6. It is the final chapter of the Dreamweaver CS6 Tutorial.