GitHub Projects

Seriously. One of the first lessons I learned in graduate school (the hard way) was that you should ALWAYS inspect your data before doing anything with it. Make sure your FITS images open fine and don't look weird, learn what the column names are in your csv table and if there are any header rows that you'll need to tell your code to ignore, make sure that you understand how to access the information in the data you have, etc.

It makes sense, right? You need to know what you're working with, need to know if there's anything wrong or anything broken. If someone shared the data with you, you need to know if there's anything you need clarification on don't understand sooner rather than later. Looking at the data first will save you so many headaches in the future. Truly.

Ways to Look at Data

• SAOimage DS9 – an astronomical imaging and data visualization GUI application. Can look at FITS images in single color, create RGB view from three FITS images, create regions that record the world coordinate system of the FITS files, calculate statistics, etc. If you take no other advice from me, astronomers of any flavor should at least install this app.
• QFitsView – similar in function to DS9 in function except it can also show 1D spectra underneath the 2D image for 3D spectroscopic data.
• TOPCAT – an interactive graphical viewer and editor for tabular data. Incredibly useful for looking at catalogs of galaxies/stars/etc. (both FITS tables and regular csv-type tables), cross-matching between catalogs using astronometry, and more.
• python + matplotlib – python is a powerful programming language, one that the majority of astronomers use to conduct their science, with the matplotlib package serving as an invaluable tool for much of that visualization and exploration. However, this combination can also be useful when first looking at the data as well! For example, when exploring a catalog of targets you can plot various galaxy properties against each other or when accessing a 1D spectrum you can plot the spectrum to look for any discernible spectral features, etc. For any of my code, you'll see so many examples of me visualizing things at each stage of analysis, both to help with workflow+analysis and also to make it easier to troubleshoot issues.

TLDR: when writing code, it's a good idea to start with the smallest test case first — once you have a solid foundation and the code is doing what it should, ONLY THEN should be the time to start scaling up to your larger vision for the code

This may seem obvious but when it comes to any big task, project, goal, etc., it's always best to start small and work in bite-sized chunks. Taking a big project and breaking it down into smaller and smaller parts ensures that you have a firm grasp of what you need to do, the steps you need to take, and a plan on how to do those steps.

This is especially important when it comes to coding! Imagine you need to analyze a lot of spectroscopic data as part of your research. The code you need to write may include reading in various spectroscopy data files, exploring them, fitting the spectral emission features, and then plotting and printing all of the results. That's a big task and it can be easy to fall into the trap of trying to write all that code at once — especially if you've done some/all of those tasks before in other scenarios. However, that's the easiest way to create a series of small mistakes within your code that compound into large issues, incorrect calculations, or possibly the code just not working at all.

Instead, if we were to start small and then scale up, it may look something like this:

Starting Small

1. Chose and read in ONE spectrum from the larger sample of data.
2. Inspect the data you've read in, making sure you have everything you need.
   • Look at the file contents
   • Identify any header information about units, WCS, etc.
   • Assuming the spectrosopic data is a table, make sure you have columns of wavelength and flux
   • Make a quick plot to look at the spectrum and note if anything looks weird
3. Identify the information you want to extract from the spectrum.
4. Set up and run fitting code to fit the strongest spectral feature from the list of spectral features you've identified; iterate until the fit results are working correctly.
5. Set up and run fitting code on the rest of the spectral features, ensuring all fit results are working correctly.
6. Calculate the information you want from the fit results.
7. Plot a final spectrum with fits and/or a table of the calculated results.

Scaling Up

1. Start by scaling up one step — in this example, that could be reading in a second spectrum and running the full code to make sure it all works on the new data. This is especially useful if the full suite of spectroscopic data come from different databases (and therefore may have different formats; i.e., table format, FITS table, etc.), where you may need to have some conditional flags that change a few ways that the data are read in or processed depending upon which database they came from.
2. Once all that works, assuming you've tested the code for each type of data (if the full dataset came from different databases), run the code on the full suite of spectra.

TLDR: do some research FIRST and follow ethical best practices... before your IP gets blocked from a website because you accidentally DOS attacked it with your web scraping code.

Web "scraping" or "crawling" describes using code of some kind to explore & extract information from a web page. This concept has existed for almost as long as the internet has, which has lead to many interesting conversations about the ethics of web scraping: how to do so respectfully, legally, and how to avoid accidentally overloading a website with requests.

In 1994, early internet users and web-scraping robot developers created a set of rules and guidelines on how to operate on the web when using robots. This consensus has since been referred to as the Robot Exclusion Protocal (or "robots.txt") and is a common standard used by web developers, and can be found on most websites on the internet. The original page created by this group, outlining the protocal, can be found at robotstxt.org/orig.

What is "robots.txt"? Essentially, this protocal recommends every website have a file on the server hosting the site, where the contents of this file describe everywhere on the site that the website owner does NOT want robots to scrape information. Everything in the "Disallow" section of a "robots.txt" file are places where no robot should visit. This doesn't mean that all robots will respect the parameters in "robots.txt", but most should at the very least. The most restrictive version of this file (that Disallows any access at all) looks like this:

User-agent: *  # this applies to all robots
Disallow: /  # all pages restricted, no robots allowed

Some Ethical Do's & Dont's

• throttle your requests — add delays or rate limits to your code, to avoid spamming a website with requests/tasks really quickly (which is essentially a DOS attack); especially important when you're in the testing or trouble-shooting phase where you're re-running code a bunch // this can be done even with just adding some seconds between each request/task!
• check for APIs — some websites have or recommend an Application Programming Interface (API) for accessing their Allowed information, use these when available or applicable to your needs because they are literally there to be reliable, respectful, and helpful // "what is an API?"
• start small, then scale up — before you go all out with your grand plan, it's a good idea to start small and test everything at that smaller scale in order to find bugs, fixes needed, and learn how to interface with that website; after that, then you can begin to scale up to the full coding program you want
• stick to what you need — when starting out with web scraping, it can be fun to gather all the information possible just because you can; but that can build the number of requests to the website (risk of overloading) without actually helping with what you're trying to do with your code // keep it simple and stick to only what you need

More on Robot Exclusion Protocal / "robots.txt"

• robots.txt // Wikipedia
• How to find the Robots.txt file? // scrape.do
• Robots Beware // an example of a Robot Exclusion Protocal with more information and background context, from arxiv.org (sidenote: this was the site I mentioned in the TLDR, thankfully it was only a 48hr ban for the first offense)

Ethical Web Scraping Practices

• "How do you make a web scraper respectful and ethical?" // OpenIndex.io
• "Ethical Web Scraping: Principles and Practices" // DataCamp
• "Ethical Web Scraping with BeautifulSoup: Best Practices and Legal Considerations" // Medium:TomTalksPython
• Ethicrawl: a python library for ethical, professional-grade web crawling (note: not yet vetted by Taylor)