Liner Notes
The FATpick Blog

Real-time performance feedback is one of FATpick's signature features.

As you play along with a song in the tablature viewer, FATpick listens in, tracking the pitch and rhythm of each note that you play. Your performance is reflected in the score - and accuracy and timing charts - that you see in the performance summary that pops up at the end of each song (or when you pause playback). You can also configure the tab viewer to give you real-time feedback your rhythm and precision as you play.

Benefits of Automatic Pitch and Rhythm Feedback

When accurately calibrated, this feedback is an invaluable tool for improving your playing:

  • "Gamification" of guitar practice with the help of a simple, consistent performance metric - reported at the end of each song and in a personalized dashboard of aggregate performance over time - is extremely motivating for many. Competition is exciting, whether you are fighting for a spot on the leaderboard or "just" to beat your personal record. And it's always satisfying - and sometimes a necessary morale boost - to have a visible reminder of just how much progress you've made.

  • When you are new to the guitar - or someone with more experience playing close to the edge of your abilities - nearly all of your attention is focused on playing. This sort of "stretching" should always be a component of your practice routine; playing just beyond your level of comfort or competence is a great way to improve. But it can be hard to really listen to how well you are playing at the same time. Whether it's just a tricky patch of a particular song or a more systematic problem you need to work on, objective feedback, provided in real-time as you play, is a powerful way to identify - and interactively address - these sorts of subtle problems.

This feature is powered by a sophisticated algorithm that the FATpick team has developed over the past 18 months (and counting). Informed by academic research and refined by relentless (and ongoing) field testing, the algorithm is designed to efficiently and accurately identify the pitch and onset/offset time of notes generated by plucked or strummed strings as recorded by diverse audio capture devices.

No special equipment is required. FATpick works with any guitar or bass (even banjo or ukulele), whether electric, acoustic or in-between. And while FATpick can take advantage of DI boxes, digital audio interfaces, Guitar-to-USB cables when they are available, something as simple as an external microphone placed in front of an amp, or even your laptop's built-in microphone is all that is needed.

What These Settings Are For

Most users don't need to worry about tweaking the audio analysis settings. The pitch and note detection algorithm works well out of the box.

But make no mistake. Digital signal processing is a complex topic in general, and support for diverse instruments, audio capture interfaces, host computers, even environmental acoustics, makes our specific application of it even more challenging.

Besides, we understand our customers. We know how frustrating it can be to know you hit that note when the computer says you didn't. You recognize how valuable the performance feedback can be and really want to take full advantage of it. But if you're going to take it seriously, you want it as close to perfect as it can be. Just like many of you, we've spent a lot of time tweaking guitar, pedal and amp settings to dial in the perfect tone. You've got high standards for this sort of thing, and are willing to invest time and energy to reach them.

For these reasons we have made a point to give users that want the opportunity to fine-tune the audio analysis algorithm the tools and knowledge to do it. That is the purpose of the Advanced Audio Analysis Settings Panel.

This is a little bit of a balancing act. The pitch and onset/offset detection algorithm is complicated, and still evolving, even now. We're trying to make the most broadly useful configuration parameters available while avoiding anything excessively complicated or difficult to adjust properly.

This means the advanced audio analysis settings that are available may change - sometimes with little or no notice - as we make adjustments to the utility/complexity balance or as the overall algorithm is refined. (You don't need to worry about your customizations "breaking" the app or the algorithm when these changes occur. The most that could happen is that customizations to "legacy" or "retired" settings will no longer have any effect at all, and that's the worst case scenario. We're just warning you that in future releases you might notice existing settings are reworked or removed, or that new settings introduced. Typically these changes will be covered in the release notes.)

How to Enable Advanced Audio Analysis Settings

Note that the Audio Analysis Settings control panel is hidden by default. Since most users will find the defaults adequate we want to prevent accidental or uninformed changes to these settings. Arbitrary changes to these settings are at least as likely to degrade this critical feature as they are to improve it.

In order to view or make changes to the Audio Analysis Settings you must first enable "Advanced Options" from the General Settings panel. To do this:

  1. Open the Settings menu by clicking on the gear icon (⚙️) in the navigation bar - or, in most contexts, by typing the letter S to invoke the equivalent keyboard shortcut. This will open the main settings menu.

  2. From that main settings menu, click the button labelled "General" (or type the letter G to invoke the corresponding keyboard shortcut) to open the General Settings.

  3. Within the general settings menu, find the toggle switch labelled "Show Advanced Options" and flip it to the enabled state.

Once the advanced options are enabled, you should immediately see several new configuration options, both on the General Settings control panel itself and in some of the other settings screens once you navigate to them. You will also see new options in the main Settings menu, including a button labelled "Audio Analysis".

Once the Advanced Options are enabled, navigate back to the main settings menu and select "Audio Analysis" by clicking the button or pressing the N key to activate the corresponding keyboard navigation shortcut.

Note that in order to maintain consistency during game play, Audio Analysis Settings cannot be changed when the tablature player or track selection screens are active. In that case the Audio Analysis button will be disabled (grayed-out) within the main Settings menu and you must close the settings dialog and navigate to the home menu (or another suitable screen) before opening the Audio Analysis Settings.

By the way, any Audio Analysis customizations you have made will remain in effect whether or not the general "Advanced Options" are enabled. That is, even if you 1) enable the advanced options, 2) make changes to the advanced audio analysis configuration, and then 3) go back to the general settings and disable "Advanced Options" so that the Audio Analysis option is no longer visible on the main Settings menu, your custom audio settings will still be active. If you don't need the advanced options enabled all the time this is a handy way to partially "lock" the audio analysis configuration to prevent accidental changes.

Don't Panic, You Can Restore the Original Settings

While it's possible to "break" or at least diminish FATpick's pitch and rhythm detection algorithm with a sub-optimal advanced Audio Analysis configuration, you should not be afraid to experiment with different settings. You can't do any permanent damage.

If you'd like to undo your customizations, whether because you think you've made things worse or you just want to roll back to the factory settings and compare, you have a few different options:

  1. You can revert the audio analysis settings - and only the audio analysis settings - back to the "factory" (default) configuration directly from the Audio Analysis Settings panel by clicking the "Restore Default Analysis Settings" button. This will immediately restore all of the advanced audio analysis options back to FATpick's default configuration.

  2. More generally you have the ability to revert ALL settings - including but not limited to the audio analysis settings - back to the default configuration by selecting "Restore Factory Settings" from the General Settings screen. This will replace all custom application settings with their default values.

  3. You also have the option to save the current application settings - or restore the most recently saved configuration - using the "Back Up Settings" actions under General Settings.

    Use the "Back Up" button to save your current application settings to the cloud. Your configuration will be uploaded to and indefinitely stored on FATpick's servers (but only your account will have access to it).

    Use the "Restore" button to revert back to the previously saved settings. This will replace all of your current settings - including but not limited to the advanced audio analysis settings - with the configuration you last backed up to the cloud.

    Note that the cloud-based application settings back-up was primarily created to make it easy to transfer app settings from one computer to another, or to easily preserve and restore settings in the event that FATpick must be fully removed and reinstalled on a single computer, such as a complete OS replacement (rather than an upgrade). The cloud back-up is also a convenient way to confidently experiment with new configurations, knowing you can easily revert back to your previous settings. But note that only the most recently backed-up configuration is saved to the cloud. You can revert back to (restore) that configuration as often as you want by clicking the "Restore" button, but clicking "Back Up" will replace the previously stored back-up. You can only save one version of the configuration at a time.

The Audio Analysis Settings

Once enabled and selected, the Audio Analysis Settings appear in a modal popup that looks something like the following screenshot.

See below for a detailed description of each control.


Advanced Audio Analysis Settings
(Click the image to view it full-size in a new window.)

Input

This section, duplicating the audio capture configuration from the generic Audio Settings control panel, is provided as a convenience. It can be used to select and configure the "input device" (microphone or other input) that FATpick's pitch and rhythm detection algorithm will listen to.

Input Device

Use this drop-down menu to select the input device that FATpick should monitor to hear your play the guitar.

For example, if you've ever tried to make a digital recording of your guitar playing, maybe you plugged a standard 1/4" guitar cable - or the cable of an XLR microphone in front of your amp - into a direct (DI) box or other computer audio interface in order to record directly into your DAW or other software. You can use the exact same setup with FATpick. Or maybe you have a 1/4"-guitar-to-USB cable, or an external USB "podcasting" microphone. FATpick also works with those. But none of that is strictly necessary. A majority of FATpick's users don't rely on anything more sophisticated than their laptop's built-in microphone, or whatever 1/8"-AUX-cable-style external mic they can hobble together if their computer didn't come with a built-in microphone. All of the currently active audio capture devices that FATpick can detect will be listed as options in the drop down menu. FATpick's audio analysis will be based on whichever device is selected.

Note that with this setting you can configure FATpick to listen to an audio source that is independent your operating system's default. That's convenient if you do have a specialized audio interface or even a dedicated external microphone. By selecting the guitar-specific input device from this drop down and leaving the general purpose mic as the system default, you can easily switch between FATpick and, say, FaceTime or Skype calls, without making any changes to your audio settings.

However, if you want FATpick to use the same mic as the rest of the programs on your computer you can do that too. When you select the input device labeled "Default" FATpick will always use whatever microphone is currently selected in your operating system's "Sound" control panel or settings.

FATpick will also switch back to the system default input device if the currently selected device is disconnected or becomes unavailable any other reason.

Input Volume

This slider controls how "loud" the input audio will seem to FATpick.

It's best in general to set the input volume to be moderately loud but not all the way up. Something like 70% to 80% is a pretty safe bet. But if it looks like the input audio is too quiet for the app to pick up cleanly - or is being clipped for being too loud - you may need to deviate from that baseline.

Note that the input volume controls found in the operating system's Sound settings (Windows) or Control Panel (Mac) is applied before and independently of this setting. That is, if input volume is turned way down at the operating system level, FATpick won't be able to hear your playing very well, no matter how high you set the volume with this slider.

As may be obvious the same statement is true for hardware-based input volume controls, if there are any in your setup. If there is a way to control the input volume on the microphone itself, or in a "mixer" or DI box or any other part the chain that the signal passes thru before it reaches the computer - that volume control will take effect before and independently of the operating system settings. If your microphone is turned down too low, your computer will not get enough signal to work with, no matter how high you set the volume at the software tier.

The advice used for the in-app volume setting also applies to the operating system, and to external hardware controls - as a general rule of thumb you should set the input volume to be moderately loud but not all the way up.

Input Volume Meter

Beneath the input volume slider you'll find a simple linear "VU level" or "loudness meter" that indicates the roughly-instantaneous strength of the signal FATpick is getting from the currently selected input device.

This meter serves two primary functions:

  1. It provides basic but immediate validation of whether (and to what extent) FATpick is able to obtain a signal from the selected input device. If the volume meter isn't showing any signs of life then something is not working properly. You should double check that everything is connected, powered on and not "muted" or turned all the way down. If the volume meter isn't working with that input device pitch and note timing detection probably won't work either.

  2. The level shown on the meter is a good way to confirm that that volume level is properly set up throughout the entire signal chain.

    The level of the volume meter should track the input volume setting pretty closely. When you turn the slider down low, the volume meter should stay pretty low as well, even when you make a load noise into the the microphone. When you turn the volume slider higher, the maximum level that appears in the volume meter should rise accordingly.

    As a general rule of thumb you want the volume meter to light up most of its box when the microphone picks up a loud noise, but you don't want the signal to max-out very often. If the volume level of the meter isn't responding to the slider as well as expected you should check the input volume settings in the rest of the signal chain.

Analysis

This sliders in this section control some of the key parameters of the core pitch and note onset/offset detection logic.

Each slider controls an independent variable, but there is some indirect interaction between the settings.

Window Size

This slider determines the width (duration) of the "sample window" that is the base unit of analysis for pitch and onset/offset detection.

Roughly speaking:

  • When the sample window is small the algorithm will have lower latency but also lower quality. Narrow sample windows let the algorithm provide feedback more "instantly" - closer to true "real-time" but can can make the algorithm less decisive. It might determine that a given sound is likely to represent one of a few different pitches, but can't choose between them with confidence.

  • When the sample window is large the algorithm will have higher latency but also higher quality. The lag between a note being played and the algorithm grading it as "hit" or "missed" becomes more noticeable, but the algorithm is also better able to distinguish between two otherwise similarly sounding tones.

but neither the latency nor the quality impact is quite as linear as those generalizations seem to imply.

There is at least one relevant, hard constraint on the sample window size that is particularly notable however:

As you probably know, the "pitch" of a sound is determined by - essentially defined as - the frequency of corresponding sound wave - the rate at which the object that is inducing the sound is vibrating. For example, when the low E string on a typical six string guitar (in the standard tuning) is plucked, it begins to vibrate at approximately 82 Hz (cycles per second) producing the pitch we call (in scientific notation) E2. This means that - if you imagine the sound as a perfect sine wave (not truly accurate, but more than adequate for our purposes here) - that means your ear is hit by a crest (peak) of the sound wave roughly 82 times per second. Inversely that means it takes about 12 milliseconds (0.012 seconds) to observe one complete cycle of the sound wave, since 1/82 is approximately 0.0122.

Since the window size literally determines the maximum slice of time that the algorithm looks at when trying to estimate the frequency (pitch) of a sound wave, that means that the duration of the sample would would need to be at least 12 ms in order to observe even a single cycle of an E2 sound wave.

When you consider that actual sound wave produced by a guitar - especially the low frequency sounds generated by fat guitar strings or virtually any string on a bass guitar - is substantially more complicated that pure sine wave, it becomes clear that it quickly becomes difficult, if not impossible, to confidently or accurately identify the pitch E2 when the duration of each sample is at or below that needed to observe even a single example of full cycle of the wave.

In other words, the duration of the sample window determines a lower bound for the deepest pitches (lowest frequencies) that can be identified with any confidence. There's no satisfactory way to detect the presence of a noisy or compound signal without the ability to observe even a single cycle of the corresponding pure tone.

As a practical matter this is a mostly hypothetical problem. Any pitch we might reasonably is detectable throughout almost any window size you can set with the existing slider. However, at the extreme high end (least duration) of the available window size scale, the lowest pitch notes typically found on a standard four string bass guitar (roughly sound at and below the note B1) cannot be reliably detected. The smallest window size settings will make it impossible to detect notes in this range.

Smoothing Factor

This slider determines the strength (magnitude) of the dampening factor that is used to smooth over rapid (high frequency) variations in the input signal.

The smoothing factor more or less smooths out short time-scale variations in the strength of the signal found in the sample window by applying a moving average to the data.

This smoothing is intended to make "noisy" data look more the pure sine waves that are expected to approximate, but requires an inherently lossy transformation. There is no way to "smooth" the data without stripping out information. We're just hoping that the information that is lost is mostly true "noise".

The smoothing factor effectively makes the input signal seem to change more slowly that it does it in reality.

When the smoothing factor is relatively small the time scale of these fluctuations is unlikely to interfere with the pitch detection logic - the data that are smoothed are unlikely to represent anything that could be successfully identified as sound intentionally generated by a guitar string.

But since the note onset and offset logic relies in part on detecting short-time-scale fluctuations in the strength (volume) of the input signal, even relatively small smoothing factor settings may make it more difficult to identify when "quiet" notes begin and end.

The smoothing factor can help the algorithm handle noisy environments (or low quality microphones) more successfully.

But as as the magnitude of the smoothing factor increases, so does the impact on the note onset detection logic. Aggressive smoothing will also interfere with the detection of high frequency (high pitch) sounds and identifying individual tones within polyphonic sounds (like strummed chords).

Min Flux Threshold

The sound wave generated by a plucked guitar string tends to follow a predictable pattern: there's a sudden, semi-chaotic spike in the strength of the signal which quickly smooths out to recognizable tone and then slowly tapers off as the sound fades away. These short-term changes in the relative "strength" (volume) of the input signal are important to FATpick's process for identifying the beginning (onset) and ending (offset) of each note that you play.

This slider influences how sensitive the note onset/offset algorithm is to changes in the strength of the input signal.

When the minimum flux threshold is set low, small changes in the strength of the input signal will be interpreted as the beginning (or end) of a note. When the minimum flux threshold is high, a larger change in dynamics is needed before the algorithm recognizes the start of a new note.

A large flux threshold is appropriate for an aggressive or dynamic style of playing, while the threshold may need to be lower for softer or mellow styles of play.

This isn't just about volume or amplification. For example, the dynamics of steel string guitars - whether electric or acoustic - or any instrument with a bright tone - lend themselves to higher flux threshold settings, while instruments with a soft or mellow tone - a nylon-string acoustic guitar certainly, but also an electric bass with flatwound strings, even when amplified - might benefit for a lower flux threshold.

When the flux threshold is set too high, FATpick might miss some of the notes you are playing - not which note you are playing, but whether you or not you are playing a note at all.

When the flux threshold is set too low, FATpick can confuse incidental sounds - background noise, natural harmonics, even the sound of your fingers sliding on the strings - with the notes you are playing, distracting the algorithm with too many false notes.

All things being equal, it's probably slightly better to err on the side of setting the minimum flux threshold too low than too high, but you should exercise some restraint. There are subtle but important interactions between the note onset/offset and pitch detection logic. If there are too many false note-onsets the algorithm will have trouble correctly identifying the pitch of the notes that you are playing.

You can use the pitch monitor - both here and on game-play (tablature viewer) screen - to check the sensitivity of the note onset detection. There's a small circle in the upper-right corner of the pitch monitor that lights up whenever the start of a new note is detected. Since the duration of an actual note onset event is tiny, you might notice some flickering as that light fades in and out, but if the note-detected light is lit too often or not often enough you may want to make changes to the audio analysis settings.

Buckets per Sample

This slider controls the resolution of note onset/offset detection, just as the "min flux threshold" slider determines its sensitivity.

If you are familiar with the concept of an ASDR envelope, that may be a helpful way to understand the effect of the min-flux-threshold and buckets-per-sample settings. The minimum-flux-threshold value is correlated with the height of the attack phase of the envelope - the maximum volume that the sound reaches. The buckets-per-sample value is correlated to the slope (or more directly, duration) of the attack phase of the envelope - how quickly that peak volume is reached.

When the resolution is high - i.e., when this slider is set to a large value - onset detection is fine-grained and better able to detect rapid fluctuations in the strength of the input signal.

When the resolution is low - i.e., when this slider is set to a smaller value - onset detection is coarse-grained and able to detect more gradual changes in the strength of the input signal.

High resolution onset detection (large values of "buckets per sample") can be more precise, but may yield false positives if the input signal is noisy, or have trouble recognizing instruments with a mellow tone due to their comparatively sluggish attack.

While far from perfect, one heuristic for setting the resolution of the note onset/offset detection algorithm is to consider the register (range in pitch) of your instrument, and whether the sounds it produces are "percussive" or "mellow"

  • Low frequency sounds with a gentle attack - like the low pitch and mellow tone associated with a bass guitar's thick and stiff strings - are at one end of the spectrum. Coarse-grained onset detection (small values of buckets-per-sample) may perform best in that scenario.

  • The high frequency and fast attack of sounds generated by elastic, high-tension strings -like the thin B and high-E strings on a six-string guitar in the standard tuning - are at the other end of the perspective. Fine-grained onset detection (large values of buckets-per-sample) may be more appropriate for these high pitch, percussive sounds.

Filter

The settings found in this section are a little different from the ones we've discussed so far.

Unlike the controls found in the "Analysis" section, which influence the pitch and note detect algorithm directly, the "filter" settings determine whether and how the input signal is shaped (transformed) before the audio analysis really begins.

You can think of the audio analysis filter sort of like a pre-amp or guitar pedal that sits between your guitar and your amplifier. When the pedal is active it will transform the audio signal (coming out of your guitar) before it even reaches the amplifier. This filter works the same way. When the filter (the pedal) is active it will transform the audio signal (received via the microphone) before it even reaches the pitch/note detection algorithm (the amplifier).

Loosely speaking the filter let you "shape" the input signal like the sliders on an equalizer: turning the volume up for a certain range of frequencies and down for others.

Apply Filter

This toggle button controls whether or not the filter is active. When the button is togged to the off position the filter is disabled and does not have any effect.

You can still make changes to the configuration of the filter while it is disabled. But - just like the knobs on an inactive guitar pedal - your changes won't take effect until you re-enable the filter.

Filter Type

This drop down controls the general class of the audio filter.

The conventional audio filter circuit types are available:

  • Low Pass - the low-pass filter will allow (pass-thru) sounds below the cut-off but "filter-out" higher frequencies. (To be more technically precise, the filter will progressively weaken (attenuate) the signal above the cut-off frequency.)

  • High Pass - the high-pass filter does the opposite, allowing frequencies above the cut-off to pass-thru but attenuating the signal below the cut-off

  • Bandpass - the bandpass filter filters out frequencies outside of a given range, sort of like the combination of a low-pass and high-pass filter.

  • Notch - the notch filter is the opposite of a band-pass filter. It attenuates frequencies within a given range, allowing lower frequency higher frequency signals to pass through.

  • Low Shelf - the low-shelf filter will boost (or cut) the signal below the cut-off frequency, while ignoring higher frequencies.

  • High Shelf - the high-shelf filter will boost (or cut) the signal above the cut-off frequency, while ignoring lower frequencies.

Frequency

This slider controls the "cut off" frequency of the pre-analysis audio filter.

The precise role of the cut-off frequency depends on the filter type, but it represents the threshold value in a general sense

For example, when a low-pass filter is used the cut-off frequency determines the maximum frequency before the filter starts to attenuate the signal.

Note that in the case of a bandpass, peaking or notch filter this frequency indicates the center frequency of the band (and the Q-factor controls the width that band).

Q-Factor

The Q-factor or quality factor is little hard to explain without delving into a lot of technical detail.

Roughly speaking the Q-factor is a measure of how quickly the signal amplitude drops off from the filter's peak (resonance) frequency.

In practice you can think of the Q-factor as a way to to control the "width" of a bandpass, notch or peaking filter (although the slider is backwards from that perspective - the larger the Q-factor the narrower the bandwidth).

Since a narrow band is desirable in most applications, the Q-factor ("quality factor") is inversely related to the width of the spike.

A large Q-factor implies a steep, narrow spike in which the amplitude of the signal drops off sharply as you move away from the center frequency.

A small Q-factor indicates a duller, more gradual transition across a wider range of frequencies.

Note that in the case of a bandpass, peaking or notch filter the Q-factor controls the width of the filter (the range of frequencies to which the filter is applied), centered around the value set by the frequency slider.

Gain

For shelf or peak filters this slider controls the degree to which the signal is boosted (or attenuated).

This setting has no effect on low-pass, band-pass, high-pass or notch filters.

Restore Defaults

Clicking the button labelled "Restore Default Analysis Settings" will reset all of the audio analysis settings back to their default values.

This will overwrite (replace) any changes you've made to the settings on this screen and restore FATpick's default ("factory") audio analysis configuration.

Live Test

Click the "Test Settings" button to enable a "live test" mode that you can use to validate your current settings, visualize the impact of different settings or to interactively experiment with new configurations.

When the test is enabled three useful visualizations will appear in this section.

  • On the left you'll find a spectrogram that charts the amplitude of the current input signal by frequency. Each vertical bar in the chart represents a narrow frequency band. The height of each bar indicates the amplitude of the input signal at that frequency.

    This visualization is particularly helpful for observing the impact of the the pre-analysis audio filter. When the filter is used to boost or attenuate the signal at different frequencies changes should be reflected on this graph.

  • On the right you'll find something that looks like an old-school oscilloscope display. This is a waveform plot that shows the (instantaneous) amplitude of the input signal over time.

    (Incidentally the impact of the Window Size and Smoothing Factor settings are readily apparent in this visualization.)

  • In the center you'll find a direct representation of the output from the audio analysis engine, the pitch monitor, which displays the note (pitch) and note onset status as reported by the pitch and onset/offset detection algorithm.

    While the monitor presents a simplified version of the algorithm's internal state, the analysis used here is exactly the same as that used within the tablature player, so the pitch monitor is strong indication of how your audio analysis settings will perform during game play.

Also see more posts by or in .
Or, go to the latest posts from the Liner Notes blog.