Blog

A blog about astronomy and software development, mainly focused on my experiences during the development of JPARSEC.

Blog

Tag index

Recent Posts

Most recent posts.

Using GPhoto for move detection

This is my first blog entry in more than two years. I've been very busy with my Android planetarium project, now almost finished, but probably needed also some rest. Despite of that during this time I have developed a number of little nice projects, and waited for the right moment to resume my blog activity and release some of them. As ussual they are not oriented to specific goals, going from 3d renderings to solving complex and time consuming tasks in a completely automatic manner, or even to criptocurrencies.

In this post I present a little program aimed to detecting move and triggering shots with a camera automatically. It is based on the gphoto library, and uses the gphoto binding implemented in the JPARSEC library. Some features of the binding have been improved or developed to support this use case. For instance, the live view mode didn't support executing other commands except those to capture previews, and it wasn't possible to execute this mode without showing a panel object with those previews.

The program provides many (31) configuration options in a file named config.txt. Among others, it is possible to set some custom parameters for the camera at startup (ISO, shutter, aperture, among others) and right before the program ends, adjust the sensitivity of the move detection algorithm, configure masks the search for move only in certain areas, or to uppload the shots to a server.

My intenction is to use this program for security, shooting when move is detected with a high resolution camera instead of a webcam. After thinking from time to time on this a saw a post in the gphoto mail list from Alan Corey, asking for a way to do something similar for wild photography. So I decided to start writing the program and Alan collaborated testing it in his cameras. In fact, the program has been tested successfully in a Canon 40D, a Nikon D5200, and a Canon Powershot S70 and A520. It has also been tested in a variety of CPU devices, from a powerful desktop to a Raspberry PI 3. Here I will describe the basics of this program, some other details can be found in the gphoto mailing list and previous messages about the same subject.

The move detection algorithm is very simple. It searches for move in two ways: detecting a global change in the luminosity of the preview image, and detecting changes in the brightness level of the pixels. The preview image is internally converted to gray scale since the color data is not required to detect move. The global change is computed from the difference between the histograms of the preview image and a reference preview image taken when the program starts. This reference image should be free of movement, and is optionally updated in a given time interval to account for possible brightness changes due to the sky or the weather. The histogram difference is computed pixel by pixel, and this change in the gray color is also used to compute the number of individual pixels that present some move or change in that level, according to a different change criteria specific to individual pixels. So the second move detection check is computed from the percentage of individual pixels showing move, and the move is considered as detected when any of those two methods trigger a detection.

When this happens the program pauses the live view in the JPARSEC binding to execute other commands to take a number of shots, resuming back after that the live view mode.

The program has as dependency the jsch library to uppload images to a server, and the itextpdf for its Base64 class used here to include images in the web server. The idea is to compile it with Java 6 at least. Here is the code of the main class.

GPhotoMoveDetection.java
import java.awt.Color;
import java.awt.Graphics2D;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.ArrayList;
 
import com.itextpdf.text.pdf.codec.Base64;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;
 
import jparsec.graph.DataSet;
import jparsec.graph.chartRendering.AWTGraphics;
import jparsec.io.ConsoleReport;
import jparsec.io.FileIO;
import jparsec.io.ReadFile;
import jparsec.io.device.implementation.GPhotoCamera;
import jparsec.io.device.implementation.GPhotoCamera.CAMERA_ID;
import jparsec.io.device.implementation.GPhotoCamera.CAMERA_PARAMETER;
import jparsec.io.image.Picture;
import jparsec.time.AstroDate;
import jparsec.vo.FTP;
 
/**
 * An example of using GPhoto library (through the binding integrated in JPARSEC) to control 
 * a DSLR camera and trigger a photo when movement is detected.
 * 
 * @author T. Alonso Albi - OAN (Spain)
 * @version 1.0
 */
public class GPhotoMoveDetection {
 
	// Export only this class to gphoto.jar and execute with:
	// java -classpath gphoto.jar:jparsec.jar:jsch-0.1.41.jar:jsch-0.1.41.jar:itextpdf-5.1.3.jar research.other.GPhotoMoveDetection 
 
	// TODO:
	// Option to pause motion detection ?
	// Server page reloading all the time, problematic with high fps. Reload by sections with different rates ?
	// Is it possible to trigger several shots quickly from shell ? Option to use the other method ?
	// Avoid overwriting old shots ?
	// Send e-mail with move alert ?
 
	public static GPhotoCamera c = null;
	public static String inISO, inAPERTURE, inCAPTURE_TARGET, inNIKON_QUALITY, inRESOLUTION, inSHUTTER_SPEED;
	public static String outISO, outAPERTURE, outCAPTURE_TARGET, outNIKON_QUALITY, outRESOLUTION, outSHUTTER_SPEED;
	public static FTP ftp = null;
	public static ArrayList<String> messages = new ArrayList<String>(), shotList = new ArrayList<String>();
	public static ArrayList<Object> thumbs = new ArrayList<Object>();
	public static int n = 0, nn = 0, moveEvents = 0;
	public static long lastMoveEvent;
	public static boolean forceExit = false, forcePause = false, createThumbs = true, keepInCamera = false;
	public static int webServerPort = 8080; // Server port (int)
	public static int liveMaxTime = 0; // sec, <= 0 => for ever
	public static int updateRef = 3600; // sec, -1 => never update
	public static int resample = 320; // width, -1 => no resample
	public static int thresholdBr = 10; // brightness % change to trigger global movement 
	public static int thresholdPx = 10; // brightness % change to trigger pixel movement 
	public static int movingPixPerc = 5; // % of pixel with change greater than threshold to trigger local movement
	public static int fps = -2; // fps of live view, < 0 for <1 fps (-2 => 0.5 fps)
	public static int nshot = 3; // number of shots to take
	public static int maxShot = 1000; // maximum number of shots to take
	public static boolean rename = true; // True to rename shots to 'shot_xxx.jpg'
	public static String sound = null;
	public static String server = null, user = null, pass = null, remoteDir = null; // to uppload images
	public static String extension = ".jpg"; // extension in lowercase for the new relevant images
	public static String mask[] = new String[] {
		"--------------------------------",
		"--------------------------------",
		"--------------------------------",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"++++++++++++++++++++++++++++++++",
		"--------------------------------",
		"--------------------------------",
		"--------------------------------",
	};
	public static boolean debug = true;
 
	/**
	 * Test program.
	 * @param args Unused.
	 */
	public static void main(String args[])
	{
		try {			
			String dir = FileIO.getWorkingDirectory();
			loadConfig(false);
 
			String out[] = GPhotoCamera.autoDetect();
			if (debug) System.out.println("Using gphoto "+GPhotoCamera.gphotoVersion);
			if (debug) System.out.println("Detected cameras:");
			ConsoleReport.stringArrayReport(out);
			c = new GPhotoCamera(CAMERA_ID.EOS40D, null, dir, false, debug);
 
			// Set custom camera parameters or read them from camera
			inISO = setParameter(inISO, CAMERA_PARAMETER.ISO);
			inAPERTURE = setParameter(inAPERTURE, CAMERA_PARAMETER.APERTURE);
			inCAPTURE_TARGET = setParameter(inCAPTURE_TARGET, CAMERA_PARAMETER.CAPTURE_TARGET);
			inNIKON_QUALITY = setParameter(inNIKON_QUALITY, CAMERA_PARAMETER.NIKON_QUALITY);
			inRESOLUTION = setParameter(inRESOLUTION, CAMERA_PARAMETER.RESOLUTION);
			inSHUTTER_SPEED = setParameter(inSHUTTER_SPEED, CAMERA_PARAMETER.SHUTTER_SPEED);
 
			c.setTimeLimitForLiveView(liveMaxTime);
			c.setLiveFPS(fps);
			c.setCopyInCamera(keepInCamera);
 
			if (debug) {
				CAMERA_PARAMETER cv[] = CAMERA_PARAMETER.values();
				for (int i=0; i<cv.length; i++)
				{
					System.out.println("Possible values of "+cv[i]);
					String values[] = c.getConfig(cv[i]);
					ConsoleReport.stringArrayReport(values);
				}
			}
 
			try {
				if (debug) System.out.println("Creating web server on port "+webServerPort);
				createServer(webServerPort);
			} catch (Exception e) {
				e.printStackTrace();
			}
 
			n = 0;
			nn = 0;
			long lastShotTime = -1;
			while(!forceExit) {
				if (forcePause) {
					if (debug) System.out.println("Live view is currently paused");					
				} else {
					if (c.isLivePaused()) {
						if (debug) System.out.println("Resuming live view");
						c.resumeLiveView();
						loadConfig(true);
					} else {
						if (debug) System.out.println("Starting live view");
						if (!c.isLive()) c.startLiveView(null, fps);
					}
				}
				long wait = (1000 / fps);
				if (fps < 0) wait = (long) (1000 / (1.0 / Math.abs(fps)));
				while (!forcePause) {
					try {
						if (forceExit) break;
						if (!c.isLive()) {
							if (debug) System.out.println("Detected live view was stopped (limit time). Waiting 5s and restarting live view mode ...");
							long t0 = System.currentTimeMillis();
							long t1 = t0 + 5000; // Wait 5s to restart live view
							while (true) {
								long t2 = System.currentTimeMillis();
								if (t2 < t1) {
									Thread.sleep(500);
									continue;
								}
								break;
							}
							String lastShot = c.getLastShotPath();
							if (lastShot != null && FileIO.exists(lastShot)) FileIO.deleteFile(lastShot);
							if (debug) System.out.println("Starting live view");
							c.startLiveView(null, fps);
						}
 
						String lastShot = c.getLastShotPath();
						boolean move = false;
						if (lastShot != null && FileIO.exists(lastShot)) {
							long time = (new File(lastShot)).lastModified();
							if (time != lastShotTime) {
								Picture pic = new Picture(lastShot);
								move = detectMove(pic);
							}
							lastShotTime = time;
						}
 
						if (move) {
							if (debug) System.out.println("Pausing live view");
							c.pauseLiveView();
							long t0 = System.currentTimeMillis();
							long t1 = t0 + wait * 2;
							while (true) {
								long t2 = System.currentTimeMillis();
								if (t2 < t1) {
									Thread.sleep(100);
									continue;
								}
								break;
							}
							break;
						}
 
						Thread.sleep(wait / 50);
					} catch (Exception exc) {
						exc.printStackTrace();
						continue;
					}
				}
 
				if (forceExit) break;
				if (forcePause) {
					if (!c.isLivePaused()) c.pauseLiveView();
					long t0 = System.currentTimeMillis();
					long t1 = t0 + wait * 2;
					while (true) {
						long t2 = System.currentTimeMillis();
						if (t2 < t1) {
							try {
								Thread.sleep(1000);
							} catch (InterruptedException e) {
								e.printStackTrace();
							}
							continue;
						}
						break;
					}
					continue;
				}
 
				// Shot/s after move detection
				moveEvents ++;
				lastMoveEvent = System.currentTimeMillis();
				addMessage("Move detected!");
				if (n > maxShot) {
					addMessage("Aborting taking shots: too much files (> "+maxShot+")");
				} else {
					// Shot inside shell using the binding
					String com = "capture-image-and-download";
					boolean lock = false;
 
					String path = "";
					for (int i=0; i<nshot; i++) {
						c.executeExternalCommandInPausedLiveView(com);
						long t0 = System.currentTimeMillis();
						while (true) {
							Thread.sleep(50);
							if (c.getExternalCommandInPausedLiveView() == null) break;
							if (System.currentTimeMillis() - t0 > 30000) {
								lock = true;
								break;
							}
						}
						if (lock) break;
						path += c.getLastShotPath()+",";
					}
					if (!path.equals("")) path = path.substring(0, path.length()-1);
 
					if (lock) addMessage("Detected camera freeze when taking shots (too much of them ?)");
					if (debug) System.out.println("Created files: "+path);
					n += nshot;
					if (!lock && !path.equals("")) {
						String p[] = DataSet.toStringArray(path, ",");
						for (int i=0; i<p.length; i++) {
							if (p[i].toLowerCase().endsWith(extension)) {
								if (debug) System.out.println("New shot found: "+p[i]);
								Picture pic = null;
								if (rename) {
									nn ++;
									pic = new Picture(p[i]);
									pic.write("shot"+nn+".jpg");
									FileIO.deleteFile(p[i]);
									p[i] = "shot"+nn+".jpg";
								}								
								if (createThumbs) {
									shotList.add(p[i]);
									if (pic == null) pic = new Picture(p[i]);
									if (resample > 0) pic.getScaledInstance(resample, 0, true);
									thumbs.add(pic.getImage());
								}
								if (ftp != null) {
									if (debug) System.out.println("Upploading "+p[i]+" to "+user+"@"+server);
									ftp.uppload(p[i], p[i]);
								}
							}
						}
					}
				}
 
				loadConfig(true);
				c.resumeLiveView();
 
				try {
					Thread.sleep(wait*2);
				} catch (Exception exc) {
					exc.printStackTrace();
				}
			}
 
			// Set at the end custom camera parameters or those read from camera at the beginning
			setParameter(inISO, outISO, CAMERA_PARAMETER.ISO);
			setParameter(inAPERTURE, outAPERTURE, CAMERA_PARAMETER.APERTURE);
			setParameter(inCAPTURE_TARGET, outCAPTURE_TARGET, CAMERA_PARAMETER.CAPTURE_TARGET);
			setParameter(inNIKON_QUALITY, outNIKON_QUALITY, CAMERA_PARAMETER.NIKON_QUALITY);
			setParameter(inRESOLUTION, outRESOLUTION, CAMERA_PARAMETER.RESOLUTION);
			setParameter(inSHUTTER_SPEED, outSHUTTER_SPEED, CAMERA_PARAMETER.SHUTTER_SPEED);
 
			if (debug) System.out.println("Exiting ...");
			if (c.isLivePaused()) c.resumeLiveView();
			c.stopLiveView();
			try {
				long wait = (1000 / fps);
				if (fps < 0) wait = (long) (1000 / (1.0 / Math.abs(fps)));
				long t0 = System.currentTimeMillis();
				long t1 = t0 + wait * 3;
				while (true) {
					long t2 = System.currentTimeMillis();
					if (t2 < t1) {
						try {
							Thread.sleep(100);
						} catch (InterruptedException e) {
							e.printStackTrace();
						}
						continue;
					}
					break;
				}
 
				System.exit(0);
			} catch (Exception exc) {
				exc.printStackTrace();
			}
 
		} catch (Exception exc)
		{
			exc.printStackTrace();
		}
	}
 
	public static String setParameter(String value, CAMERA_PARAMETER p) throws Exception {
		if (c == null || value == null) return value;
		if (value.equals("null")) {
			value = c.getParameterFromCamera(p);
		} else {
			c.setParameter(p, value);
		}
		return value;
	}
 
	public static void setParameter(String valueIn, String valueOut, CAMERA_PARAMETER p) throws Exception {
		if (c == null || valueIn == null || valueOut == null) return;
		if (valueOut.equals("null")) {
			c.setParameter(p, valueIn);
		} else {
			c.setParameter(p, valueOut);
		}
	}
 
	public static void loadConfig(boolean reload) throws Exception {
		String dir = FileIO.getWorkingDirectory(), config = dir + "config.txt";
		if (!FileIO.exists(config) && !reload) {
			System.out.println("Cannot find config file "+config);
			System.exit(0);
		}
 
		String d[] = DataSet.arrayListToStringArray(ReadFile.readAnyExternalFile(config));
		String param[] = new String[] {
			"UPDATE_REF", "RESAMPLE", "THRESHOLD_BR", "THRESHOLD_PX", "MOVING_PIX_PER", "FPS", 
			"NSHOTS", "MAX_SHOTS", "EXTENSION", "MASK", "RENAME", "SERVER", "USER", "PASSW", 
			"REMOTE_DIR", "LIVE_MAX_TIME", "WEB_SERVER_PORT", "CREATE_THUMBS", "KEEP_IN_CAMERA",
			"ISO_IN", "APERTURE_IN", "CAPTURE_TARGET_IN", "NIKON_QUALITY_IN", "RESOLUTION_IN", 
			"SHUTTER_SPEED_IN", "ISO_OUT", "APERTURE_OUT", "CAPTURE_TARGET_OUT", "NIKON_QUALITY_OUT", 
			"RESOLUTION_OUT", "SHUTTER_SPEED_OUT"
 
		};
		for (int i=0; i<d.length; i++) {
			for (int j=0; j<param.length; j++) {
				if (d[i].startsWith(param[j])) {
					if (debug) System.out.println("Reading "+d[i]);
					String val = d[i].substring(d[i].indexOf(" ")).trim();
					if (val.indexOf("//") > 0) val = val.substring(0, val.indexOf("//")).trim();
 
					if (j == 0) updateRef = Integer.parseInt(val);
					if (j == 1) resample = Integer.parseInt(val);
					if (j == 2) thresholdBr = Integer.parseInt(val);
					if (j == 3) thresholdPx = Integer.parseInt(val);
					if (j == 4) movingPixPerc = Integer.parseInt(val);
					if (j == 5) fps = Integer.parseInt(val);
					if (j == 6) nshot = Integer.parseInt(val);
					if (j == 7) maxShot = Integer.parseInt(val);
					if (j == 8) extension = val;
					if (j == 9) mask = DataSet.getSubArray(d, i+1, d.length-1);
					if (j == 10) rename = Boolean.parseBoolean(val);
					if (j == 11) server = val;
					if (j == 12) user = val;
					if (j == 13) pass = val;
					if (j == 14) remoteDir = val;
					if (j == 15) liveMaxTime = Integer.parseInt(val);
					if (j == 16) webServerPort = Integer.parseInt(val);
					if (j == 17) createThumbs = Boolean.parseBoolean(val);
					if (j == 18) keepInCamera = Boolean.parseBoolean(val);
 
					if (reload) continue;
					if (j == 19) inISO = val;
					if (j == 20) inAPERTURE = val;
					if (j == 21) inCAPTURE_TARGET = val;
					if (j == 22) inNIKON_QUALITY = val;
					if (j == 23) inRESOLUTION = val;
					if (j == 24) inSHUTTER_SPEED = val;
					if (j == 25) outISO = val;
					if (j == 26) outAPERTURE = val;
					if (j == 27) outCAPTURE_TARGET = val;
					if (j == 28) outNIKON_QUALITY = val;
					if (j == 29) outRESOLUTION = val;
					if (j == 30) outSHUTTER_SPEED = val;
				}
			}
		}
 
		if (ftp != null) ftp.disconnect();
		ftp = null;
		if (server != null && !server.equals("null") && user != null && !user.equals("null") 
				&& pass != null && !pass.equals("null")) 
			ftp = new FTP(server, user, pass);
		if (remoteDir != null && !remoteDir.equals("null")) ftp.changeDirectory(remoteDir);
 
		if (reload) addMessage("Configuration reloaded");
		if (c != null) {
			c.setLiveFPS(fps);
			c.setTimeLimitForLiveView(liveMaxTime);
			c.setCopyInCamera(keepInCamera);
		}
 
		String pp = dir + "sound.mp3";
		sound = null;
		if (FileIO.exists(pp)) {
			byte[] bytes = Files.readAllBytes(Paths.get(pp));
			sound = Base64.encodeBytes(bytes); 
			sound = "data:audio/mp3;base64,"+sound;
		}
	}
 
	public static Picture ref = null, lastPicColor = null, refColor = null;
	public static long lastRef = 0;
	public static boolean detectMove(Picture pic) throws Exception {
		if (resample > 0) pic.getScaledInstance(resample, 0, true);
 
		// Set reference at startup
		BufferedImage copy = Picture.copy(pic.getImage());
		if (refColor == null) {
			refColor = new Picture(copy);
			Graphics2D g2 = refColor.getImage().createGraphics();
			AWTGraphics.enableAntialiasing(g2);
			float fs = 20;
			g2.setColor(Color.RED);
			g2.setFont(g2.getFont().deriveFont(fs));
			g2.drawString(addMessage(null), fs, refColor.getHeight()-fs/2);
			g2.setColor(Color.WHITE);
			for (int y=0; y<pic.getHeight(); y++) {
				int cy = (int) (0.5 + (mask.length - 1.0) * (y / (pic.getHeight() - 1.0)));
				for (int x=0; x<pic.getWidth(); x++) {
					int cx = (int) (0.5 + (mask[cy].length() - 1.0) * (x / (pic.getWidth() - 1.0)));
					if (mask[cy].substring(cx, cx+1).equals("-"))
						continue;
					if (x % 3 != 0 || y % 3 != 0) continue;
					g2.drawLine(x-1, y, x+1, y);
					g2.drawLine(x, y+1, x, y-1);
				}
			}
			g2.dispose();
 
			pic.toGrayScale();
			ref = pic;
			lastRef = System.currentTimeMillis();
			addMessage("Reference created");
			return false;
		}
 
		// Detect movement
		lastPicColor = new Picture(pic.getImage());
		Graphics2D g2 = lastPicColor.getImage().createGraphics();
		AWTGraphics.enableAntialiasing(g2);
		float fs = 20;
		g2.setColor(Color.RED);
		g2.setFont(g2.getFont().deriveFont(fs));
		g2.drawString(addMessage(null), fs, refColor.getHeight()-fs/2);
		g2.setColor(Color.WHITE);
 
		pic.toGrayScale();
		boolean move = false;
		double histo[] = new double[255];
		int movingPixels = 0, maskedPixels = 0;		
		for (int y=0; y<pic.getHeight(); y++) {
			int cy = (int) (0.5 + (mask.length - 1.0) * (y / (pic.getHeight() - 1.0)));
			for (int x=0; x<pic.getWidth(); x++) {
				int cx = (int) (0.5 + (mask[cy].length() - 1.0) * (x / (pic.getWidth() - 1.0)));
				if (mask[cy].substring(cx, cx+1).equals("-")) {
					maskedPixels ++;
					continue;
				}
 
				Color c1 = pic.getColorAt(x, y);
				Color c2 = ref.getColorAt(x, y);
				int gray1 = c1.getRGB() & 255;
				int gray2 = c2.getRGB() & 255;
				int g = Math.abs(gray1 - gray2);
				histo[g] ++;
				if (g > thresholdPx*2.55) {
					movingPixels ++;
					if (x % 3 != 0 || y % 3 != 0) continue;
					g2.drawLine(x-1, y, x+1, y);
					g2.drawLine(x, y+1, x, y-1);
				}
			}			
		}
		g2.dispose();
		int maxIndex = (int) (DataSet.getIndexOfMaximum(histo) / 2.55);
		double percMov = movingPixels * 100.0 / (double) (pic.getWidth() * pic.getHeight() - maskedPixels);
		if (maxIndex > thresholdBr || percMov > movingPixPerc) move = true;
		if (debug) System.out.println("Global move: "+maxIndex+"/"+thresholdBr+". Local move: "+(float)percMov+"/"+movingPixPerc);
 
		// Update reference if required
		double elapsed = (System.currentTimeMillis() - lastRef) * 0.001;
		if (!move && updateRef > 0 && elapsed > updateRef) {
			ref = pic;
			refColor = new Picture(copy);
			lastRef = System.currentTimeMillis();			
 
			g2 = refColor.getImage().createGraphics();
			AWTGraphics.enableAntialiasing(g2);
			fs = 20;
			g2.setColor(Color.RED);
			g2.setFont(g2.getFont().deriveFont(fs));
			g2.drawString(addMessage(null), fs, refColor.getHeight()-fs/2);
			g2.setColor(Color.WHITE);
			for (int y=0; y<pic.getHeight(); y++) {
				int cy = (int) (0.5 + (mask.length - 1.0) * (y / (pic.getHeight() - 1.0)));
				for (int x=0; x<pic.getWidth(); x++) {
					int cx = (int) (0.5 + (mask[cy].length() - 1.0) * (x / (pic.getWidth() - 1.0)));
					if (mask[cy].substring(cx, cx+1).equals("-"))
						continue;
					if (x % 3 != 0 || y % 3 != 0) continue;
					g2.drawLine(x-1, y, x+1, y);
					g2.drawLine(x, y+1, x, y-1);
				}
			}
 
			addMessage("Reference updated");
		}
		return move;
	}
 
	public static String addMessage(String msg) {
		AstroDate astro = new AstroDate();
		String m = astro.toString();
		if (msg != null) {
			m += ": "+msg;
			messages.add(0, m);
 
			if (debug) System.out.println(msg);
		}
		return m;
	}
 
	public static String commands[] = new String[] {
		"Pause,left,Pausing web server",
		"Resume,center,Resuming web server",
		"Exit,right,Exiting"
	};
	public static String imgFormat = "jpg"; // jpg to reduce file size
	public static void createServer(int port) throws Exception {
        HttpServer server = HttpServer.create(new InetSocketAddress(port), 0);
        server.createContext("/", new MyHandler(-1));
 
        for (int i=0; i<commands.length; i++) {
        	String name = "com"+i;
 
            server.createContext("/"+name, new MyHandler(i));
        }
 
        server.setExecutor(null); // creates a default executor
        server.start();
	}
 
   static class MyHandler implements HttpHandler {
    	int index = -1;
 
    	public MyHandler() {
		}
 
    	public MyHandler(int i) {
    		this.index = i;
		}
 
        @Override
        public void handle(HttpExchange t) throws IOException {
        	StringBuffer response = new StringBuffer();
        	String sep = "<BR><BR>", step = FileIO.getLineSeparator();
 
			long wait = (1000 / fps);
			if (fps < 0) wait = (long) (1000 / (1.0 / Math.abs(fps)));
			int sec = (int) (wait / 1000 + 0.5);
    		response.append("<html><head><title>Web server</title><meta http-equiv=\"refresh\" content=\""+sec+", url=/\"></head><body bgcolor=\"#000000\">");
    		response.append("<center><H1 style=\"color: white\">GPhoto web server</H1></center>"+ step);
    		try {
    			response.append("<p style=\"color: white; font-size:30px; float: left\">Reference</p><p style=\"color: white; font-size:30px; float: right\">Live view (last image)</p>" + sep + step);
    			response.append("<div style=\"display: inline-block\">" + step);
        		if (refColor != null)
        			response.append("<img src=\""+refColor.imageToString(imgFormat, true)+"\" style=\"width: 49%; float: left\" />" + step);
        		if (lastPicColor != null)
        			response.append("<img src=\""+lastPicColor.imageToString(imgFormat, true)+"\" style=\"width: 49%; float: right\" />"+step);
        		response.append("</div>" + step);
    		} catch (Exception exc) {
    			exc.printStackTrace();
    		}
    		response.append(sep + "<HR>" + step);
    		if (c != null) {
    			response.append("<p style=\"color: white; font-size:20px\">Camera: "+c.getModel()+", port "+c.getPort()+"</p>"+step);
    			response.append("<p style=\"color: white; font-size:20px\">Working directory: "+c.getWorkingDirectory()+"</p>"+step);
        		String status = "running";
        		if (c.isLivePaused()) status = "paused";
    			response.append("<p style=\"color: white; font-size:20px\">Status: "+status+"</p>"+step);
    			response.append("<p style=\"color: white; font-size:20px\">Move events: "+moveEvents+"</p>"+step);
    			response.append("<p style=\"color: white; font-size:20px\">Shots taken: "+n+"</p>"+step);
    			String config = "liveMaxTime="+liveMaxTime+", ";
    			config += "updateRef="+updateRef+", ";
    			config += "resample="+resample+", ";
    			config += "thresholdBr="+thresholdBr+", ";
    			config += "thresholdPx="+thresholdPx+", ";
    			config += "movingPixPerc="+movingPixPerc+", ";
    			config += "fps="+fps+", ";
    			config += "nshot="+nshot+", ";
    			config += "maxShot="+maxShot+", ";
    			config += "rename="+rename+", ";
    			config += "createThumbs="+createThumbs+", ";
    			config += "keepInCamera="+keepInCamera+", ";
    			config += "server="+user+"@"+server+", ";
    			response.append("<p style=\"color: white; font-size:20px\">Configuration: "+config+"</p>"+step);
    		}
 
    		response.append("<HR>" + sep + step);
    		response.append("<div style=\"display: block; text-align: center\">" + step);
    		for (int i=0; i<3; i++) {
    			String title = FileIO.getField(1, commands[i], ",", false);
    			String align = FileIO.getField(2, commands[i], ",", false);
    			response.append("<input type=\"button\" style=\"color: white; float: "+align+"; font-size:40px\" onclick=\"location.href='/com"+i+"';\" value=\""+title+"\" />" + step);
    		}
    		response.append("</div>" + step);
 
    		if (index >= 0) {
    			String msg = FileIO.getRestAfterField(2, commands[index], ",", false).trim();
        		response.append("<center><H2 style=\"color: white\">" + msg + sep + step);
        		response.append("</H2></center>");
 
        		if (index == 0) {
        			forcePause = true;
        			addMessage("Paused");
        		}
        		if (index == 1) {
        			forcePause = false;
        			addMessage("Resumed");
        		}
        		if (index == 2) {
        			forceExit = true;
        			addMessage("Exited");
        		}
    		}
 
    		if (shotList != null && shotList.size() > 0) {
    			response.append("<HR>" + step);
 
    			try {
        			boolean left = true;
	        		for (int i=shotList.size()-1; i>=0; i--) {
	        			if (left) response.append("<div style=\"display: inline-block\">" + step);		        			
	        			Picture pic = new Picture((BufferedImage) thumbs.get(i));
	        			response.append("<img src=\""+pic.imageToString(imgFormat, true)+"\" style=\"width: 49%; float: "+(left ? "left" : "right")+"\" />" + step);
	        			if (!left) {
	        				response.append("</div>" + step);
	            			response.append("<p style=\"color: white; font-size:24px; float: left\">"+shotList.get(i+1)+"</p><p style=\"color: white; font-size:24px; float: right\">"+shotList.get(i)+"</p>" + step);
	        			} else {
	        				if (i == 0)
		            			response.append("<p style=\"color: white; font-size:24px; float: left\">"+shotList.get(i)+"</p>" + step);
	        			}
	        			left = !left;
	        		}
    			} catch (Exception exc) {
    				exc.printStackTrace();
    			}
 
    			response.append(sep + sep + sep + sep + sep);
    		}  
 
    		if (messages != null && messages.size() > 0) {
        		response.append("<HR>" + step);
 
        		String m[] = DataSet.arrayListToStringArray(messages);
        		for (int i=0; i<m.length; i++) {
        			response.append("<p style=\"color: white\"; font-size:16px>"+m[i]+"</p>" + step);
        		}
    		}
 
    		if (sound != null && lastMoveEvent > 0) {
    			double elapsed = (System.currentTimeMillis() - lastMoveEvent) * 0.001;
    			if (elapsed < sec) {
		    		response.append("<audio controls=\"controls\" autobuffer=\"autobuffer\" autoplay=\"autoplay\">"+step);
		    		response.append("<source src=\""+sound+"\" />"+step);
		    		response.append("</audio>"+step);
    			}
    		}
 
    		response.append("</body></html>");
 
            t.sendResponseHeaders(200, response.length());
            OutputStream os = t.getResponseBody();
            os.write(response.toString().getBytes());
            os.close();
        }
    }
}

The program includes a little web server to see the reference image with the move detection mask, the latest live view image showing the pixels with movement, some buttons to pause or stop the program, and some log messages. The server also shows the shots taken when move was detected. The next image shows how the web server looks like.


 

To access the web server in the same computer in which gphoto is running and the camera is connected you have to open a browser and load http://localhost:port, where port is the port number configured in the program (8080 by default). Of course knowing the computer IP you can load it from another device in the same network, or even from another place outside that network with a proper configuration of the router.

The full working program can be downloaded from this url. It should work with no modification in Linux and Mac. In Windows the launching script would need some modification (mainly change ; by : and write it to a .bat file), but anyway you first need to compile and install the gphoto library, so anyone capable of doing that would have no problem executing the program.

2018/01/19 13:32 · Tomás Alonso Albi · 0 Comments

ClearSky for Android released

After 4 years developing from time to time a planetarium for Android (since the first experiments with this platform back in 2012), I have published the ClearSky planetarium in two versions: a free version quite generous in features and a paid version which is oriented to become useful to amateur astronomers with telescopes. This second page contains a detailed list of features of the commercial version, although the help document included in ClearSky describes everything with even more detail.

The free version contains almost everything a casual observer would need, including the possibility of showing comets and asteroids, which is usually offered only in commercial programs in Android. But the main difference from other tools is the great accuracy in JPARSEC, superior to most free and commercial programs available even on PC platform. In addition, Spanish and English are supported, with absolutely no adds.


 

Design and user interface

One of my main concerns when developing ClearSky has been to be as objective as possible, so I haven't taken any other Android planetarium as reference to think about the design or the features. After installing all other free planetariums I think most of them are not really helpful for an amateur astronomer, and even some commercial ones (I admit I haven't paid for any of them, I prefer to enjoy developing my own one) seems to have just 'more options' or objects in the paid version, instead of being focused to 'more activities' like observing with telescopes or planning observations with a list of astronomical events or objects. Most of the development comes from JPARSEC, even in terms of design of the different color squemes, and I have developed this library for years, so this Android planetarium has the benefit of years of experience with many little cosmetic and usability improvements. However, Android development is hard, you have to hit your head against hundreds of walls and overtake all problems until you end up with a finished product.

My point of view for an adequate planetarium for general public is a program that must be extremely easy and confortable to use. I have seen too many planetariums with great graphics, but where it is hard to drag and zoom the sky, and the objects are moving everytime. This seems to be a must for other developers just to aparently mimic the natural sky, but for astronomy you need to take some time to do things and you won't want to have everything moving. Despite this, there are programs reasonably well solved in this sense, for instance the Cosmos Celestron Navigator, but it is still hard to select a body and zoom in/out. In ClearSky the sky is updated in regular intervals, and zoom operations are fast and realiable, a simple click with a finger will identify an object and will show the distance respect the previous body identified, and a double click will center that body without the mess of a menu for just that (which in ClearSky is triggered with a long press offering, among others, the options of details of the object and to track it). An example of less well solved is SkEye, where the zoom and rotation operations will mess everything, making impossible to zoom in a given body. In addition, due to rotation and the use of equatorial positions you don't know where is up, where is the horizon, or the azimuth/elevation direction, and there are too many numbers which are not important for a user. SkEye is surprisingly well valorated despite all this and the fact that I've seen three degrees of error in the position of Jupiter, so I won't call it accurate… ClearSky, although it is not as beautiful as other planetariums (although I think well enough worked in that sense), it has a much more useful user interface, making the program a really useful and confortable tool, not a simple toy to play a few minutes with. For instance, you can directly change from drag to zoom or the opposite all the time (keeping always at least one finger on the screen), something not possible in other planetariums.


 

Another issue for me is to make the user interface reasonably beautiful and confortable to the eye. This means using the adequate number and distribution of options and adequate icons. In the main window the icons have colors, and there are six for completely different tasks. SkEye shows 9!, which are too many for a phone and some of them are related to the same thing, like changing simulation conditions or the aspect, things that are not oriented to 'activities'. In ClearSky you have a search button which is a must, an option to change the color squeme which is just useful because many users expect and will play with that, and a help/trivia/more options icon (it is configurable) on the right corner of the first row (easier to click with the finger). The trivia is a game to play, just funny if you play it for a few minutes from time to time. I think it is important to have at least one configurable option for the user. The second row shows the view mode (live, text, augmented reality, and chart modes), the list of astronomical events, and the configuration option, again easier to click in that position. Since in the free version there are only two modes (live and chart), the first option will simply swap both of them, in the paid version there is a menu inevitably. The astronomical events is another must in my opinion, since a user will want to know what's going on in the sky, and if the full moon or the astronomical twilight will limit the time interval of really dark skies. All configuration options are provided through the configuration option, although the most important of them are provided in the configurable option, when it is configured to show a menu called 'more options'. The number of configurable options are a lot, almost 100, but they are properly categorized and offered in two levels, showing by default less options. The interfaz section has an option to change for a simple to a complete user interface, showing all posible options.


 

Text mode

The text mode is something probably missed in any other Android planetarium. It is only available in the commercial version, since this feature is specific to amateur astronomers, not the general public. It lists all deep sky objects and the main double/variable stars, sorted by name, object type, magnitude, position in the sky, or transit time. Supernovae and novae will also appear if you enable them, but only in case they are visible in the sky also (field of view around 50 degrees or lower). There is an option to set an alarm for the transit time (as in the events obviously), in case you want to observe some bodies in their greatest elevations. You can set an object as reference, so that the azimuth/elevation position (or right ascension/declination in case you prefer equatorial coordinates in the configuration) will appear as offsets respect that reference body. The position column will change to distance, so you can sort objects by their distances to the reference one. Very useful to observe interesting objects close to the one you are currently observing. In case you go to the text mode from the live view mode, the text mode will be also 'live', sorting the objects by default respect their distances to the direction the device points to. If you hold the device adequately on top of the telescope, you can use this feature to convert any telescope into a push-to one. In case the device doesn't point perfectly to the object, there is an align option in this case, replacing the option to set an object as reference. In case you just want a few objects you like you can add objects to a list of favourites and show only them.


 

Astronomical events

The list of astronomical events is useful to prepare an observation night. In addition to the main general events offered in the free version, the paid version includes events related to natural satellites (main Jupiter, but also for Saturn and Uranus). The events for natural satellites will show even mutual events of natural satellites (for instance a moon of Jupiter occulting or eclipsing another moon), something very interesting for amateur astronomers and probably only offered in ClearSky because of its great accuracy. The list of events for artificial satellites will show the next transits of the main satellites (ISS, HST, and Tiangong 1), as well as their transits on top of the solar and lunar disks. Iridium flares are also computed and simulated.


 

Astronomical equipment

The commercial version of ClearSky can show the field of view of any telescope with horizontal or equatorial mounts. When a telescope is selected, a long click on a star will add another option in the context menu allowing to test the polar alignment of that equipment in that star and in that moment. The program asks for four values (see documentation) and will compute from the deviations of the star (measured in pixels in the camera) where the mount is really pointing to, so that an incremental correction to improve the alignment is possible. The feature is not completely in its final status, but works.


 

Other features

The links provided at the top will list most of the features of ClearSky, but I would like to emphasize some of them. First, the catalog of stars, and specially deep sky objects, are very robust. I worked myself on the deep sky catalog for years, fixing and improving things with time and checking coordinates with Simbad (it is based on the revised NGC).

You can reach magnitude 16 in the commercial programs with the only condition of having network connection (previously downloaded fields can be used offline too). Other programs requires 1 GB of star data in your device, which is absurd for a phone. It is something spectacular to resolve globular clusters like M13 in stars, even without deep sky textures, and how star positions and textures match completely.

700 textures of deep sky objects are overlaided on the sky with great accuracy, corrected by precesion and nutation. The commercial version has high resolution textures with the possibility of downloading more.

Visual quality and accuracy are worked to a great level, so that accurate and realistic planetary rendering is possible in ClearSky. It is not as fast as I would like, but good enough (illumination and everything is done pixel by pixel without 3d OpenGL). You can even identify planetary features when clicking with the finger, something in fact used to allow simulating the sky from other bodies.


 

There are also useful color squemes, like these two screenshots show:


 

And if you are curious about the trivia, here are two screenshots of it. There are more than 200 different questions, implemented in Spanish and English, although many of them are related to identifying objects or constellations.


 

Accuracy

I will not add more words about accuracy, I have already talked about that before, comparing JPARSEC with other PC programs. In ClearSky the accuracy means this program is suitable to studies in the field of ancient arqueoastronomy, since proper motions of stars (the natural change in the shape of constellations) are considered, and planets will appear up to year 3000 B.C. Obviously more accuracy means more complicated and slower algorithms, and this is less compatible with showing the sky in real time (with objects moving), specially in a platform with strong memory and speed limitations. I don't like it, but I admit it can be required to track artificial satellites which moves fast. I currently have a list of a few bugs I have to correct, and some features which will be implemented with time to end up with a product of my (and hopefully also others) like. The hard work is done, but this is just the first release.

2015/12/24 12:48 · Tomás Alonso Albi

Astronomical Trivia

I'm currently very busy with different things in parallel, it seems uncertain when I would finish each of them. I've done a review in the last days about all of them as a frame to a talk this evening at Agrupación Astronómica de Madrid (AAM), the main astronomical amateur group in Madrid. The talk is about a tool to simulate astronomical events as they would be visible through a given kind of telescope, to prepare observations (in the science section there is a link to this presentation, in Spanish). This tool is currently limited to fellows at AAM, but could be opened to general public in a near future. Among others, I will talk about the recent esthetic upgrade to the ephemerides server, the project to create an automated observatory (more on this not before the end of the summer…), the Android planetarium (current beta available at the main JPARSEC page is quite good already, although with some bugs), and different minor things done during the years. Most, if not all of them, have been documented more or less in this blog or other pages (projects page for instance), but for one of them there is nothing about it here: the astronomical trivia.

I created a little astronomical trivia for a talk last november in the frame of the Semana de la Ciencia in Madrid, which is a two week event with lots of scientific talks about many different areas organized between many institutes in Madrid. As an experiment, I decided to do this little trivia game to let the public enjoy the 20 minutes of time before starting, when the people is still coming. I wanted to see if this thing could be or not attractive to give some fun to the people waiting, and also to prepare their minds with questions related to the talk.


 

The result was indeed quite positive. I saw all kind of faces, from people knowing everything to others showing just the opposite very clearly, and others simply laughing. Of course, there is always someone that just didn't notice at all that a trivia game was in the screen…

The program is available for downloading at this url. It should work on all operating systems (with Java installed), and in fact includes a .sh file to run it on Linux and Mac, and a .bat for Windows. A text file is used to configure everything (including the background images and the questions themselves) using a strict or rigid format, but anyway quite clear I think. The trivia contains 40 questions translated both to Spanish and to English, with different difficulty levels. Since it is fully customizable, you can replace everything to create a trivia game about any other scientific field. Windows requires a special file to account for the different carriage return character in this system, so in case you use it be careful which file you modify and in which system.

I plan to add something similar to the Android planetarium…

2015/04/07 15:51 · Tomás Alonso Albi

Recentering the disk of the Sun or Moon in eclipses

We recently (March 20) had a partial solar eclipse. At OAN we have very limited resources to do visual observing and to try to show a live view of an eclipse to the public. This time we bought a Celestron zoom ocular 8-24 mm, that includes a T2 mount for DSLR cameras, and T2 adapters for Canon and Nikon cameras. We already have a Coronado H∝ telescope, and among the staff at OAN we own different Canon and Nikon DSLRs. The idea was to show a live view of the eclipse through our eclipse web page, but weather conditions in Spain were horrible that day and it was impossible. Anyway, it was really nice to see the great interest of the people in the eclipse, and our web page became possibly the best of all Spanish web pages dedicated to the eclipse, reaching 75 000 visits the day of the eclipse and more than 200 000 in a few weeks around the date of the eclipse.

One of the main problems when showing a live view of the Sun was to recenter and crop the disk of the Sun, so that the disk seems static to the people watching it in the web page. For this task I prepared a little program to recenter and crop it, with many options like adding a few labels and upploading it directly to the server.

The program uses a simple algorithm to calculate the smallest enclosing circle for a set of points. The set of points are computed from the photo, taking the edges of the Sun (or Moon) disk and separating them from the background. Background separation is done in a very simplified way, good enough for eclipses, but not for other uses. In fact, since weather was poor a few tests that were possible showed that this separation did not work fine, since there was scattered light produced by the clouds and it was difficult for the algorithm to separate disk and background. In sunny conditions (with a background dark enough) the process worked as expected in the tests performed days before the eclipse. The algorithm, in fact, seems quite robust. The only weak point is that all points computed at the edge of the disk must be in fact there, even with one wrong point among 1000 correct ones the cropped image can be different from the expected one.


 

Previous image shows the before and after result of applying this process to an H∝ imagen taken with a Nikon camera. We did tests using both real H∝ and simulated images (using as reference an H∝ image, and also optical ones). Respect the simulated images, we tested this algorithm against a number of situations, from solar to lunar eclipses, different zoom scales, having the Sun disk partially out of the frame, and with/without background stars. All tests showed good results in the recentering algorithm, which allowed to obtain a video with the sequence of these eclipses. The images for these tests, with the Sun and Moon disk displaced respect the center, were generated using the JPARSEC library.

The following videos show the results of applying this program to the simulated sequences of the solar and lunar eclipses of 2015. Click on the links below the video frame to show each of them. The lunar eclipse sequence is created using the debug mode of the program, that shows the set of points around the edge of the disk used to later get the enclosing circle. As you can see, the algorithm is quite fast, processing a 10Mpx image in 0.5 seconds or less in modern PCs (with the debug mode activated the process is slower).

Solar eclipse H∝ / Lunar eclipse

The program is available for downloading at this url. It should work on all operating systems. A recenter.txt file is used for configuring the different options of the program, like input/output image, labels, the border to leave around the cropped image, file upploading, and many others. Two nice features are to obtain the date of the photo from the EXIF data of the JPG file using this library, and to uppload it directly to a server using the Jsch library. The hot pixel parameter controls the maximum size a star or image defect can have so that artifacts with a size less than this are not considered to lie at the edge of the disk.

Fortunately there is a transit of Mercury next year, so this work can still be of use. I also expect it to be useful for others with the same problem.

2015/03/21 12:48 · Tomás Alonso Albi

Things to fix after installing Linux

I started using Linux more than 10 years ago, and since 5 or so it is the only OS I use. It is a great OS for almost everything (now even gaming), and recently I helped my sister to recover a lot of files from a damaged USB disk that Windows won't even show as an available device.

I started my Linux experience with Mandriva and migrated later to deb based distributions like Kubuntu. At work we have Debian with KDE, and I have always been more confortable using KDE than GNOME or other window managers. I would say the best Linux app I've ever used is Konqueror (file manager, web browser, image/pdf visualization, sftp sessions, profiles to save your tabs, …), and the best Linux OS I have ever worked on is Mepis 8, which is like Debian stable with KDE 3, but with some additional tools and updated software and kernel (for that time). Mepis 11 was also great (my father still uses it in a PC conected to the TV!), but now it is not maintained anymore. An alternative is the antiX distribution, which is also very good.

Recently I bought a Dell laptop and installed Kubuntu 14.04 on it. There are so many Linux distributions that many would say they prefer a different one. Mint is now the favorite of many, but I don't like the fact that it is based on Ubuntu (with some cosmetic improvements you can also get on Ubuntu). Since Ubuntu is now going to implement new projects it is not clear to me what will happen to Mint. Debian continues to be very hard to install, in fact in my new laptop it does not detect Ethernet nor wifi. That's why antiX (MX-14, see mepis.org website) is so useful to install a Debian rolling system with a reasonable up-to-date kernel (= hardware support), and it includes a one-click installation option for KDE and even GNOME and other window managers.

Anyway, the point is that I have suffered a lot a problems with Kubuntu in the last weeks, forcing me to surface the web to investigate how to solve them. There are always some tweaks required with some apps to make them work as you want (or to make them work at all), but this time the amount of tweaks are excesive and probably many people facing at Linux for the first time would think Linux is a bad OS (and certainly is getting so). It used to be fast and easy to configure, but now seems too many new software is released (who knows how many active distributions or window managers are there?) but the old good one not properly maintained. I thought all or most of these problems were specific to (K)ubuntu, but after installing other distributions I found the same problems, coming directly from the 'stable' branch of Debian.

If you surface the web you will easily find many pages warning about bugs in the 14.04 family of Ubuntu, something you will not find so easily with previous releases. As those people, I would like to share here all the tweaks and configuration changes I have done to my Kubuntu 14.04 to finally obtain a stable system running far better than any Windows system could ever run. I don't mean installing general packages, enabling repositories, or configuring the desktop with wallpapers or screenlets in the background. I mean installing or fixing things to make everything usable. Of course, many of these steps are only required when you use an important amount of packages, like me, many normal users would feel happy just fixing a couple of things. However, it seems to me there are too many steps this time…

Additional repositories

There's some additional software I always want to have in my PC, mainly for gaming and for Java development (without the always problematic and almost impossible to keep away OpenJDK). I have added to my usual list the repositories of vlc and handbrake, since the default packages in Kubuntu 14.04 have frequent crashes and the latest version fixes them. Latest vlc version also supports the subtitles plugin not available in the 2.1 version. Gimp is also new in my list, since I would like to test the 16 bit per channel support when it is available. There was a bug also in the original MythTV package (can't remember what bug) so I added the mythbuntu repository and updated MythTV as well, which raises new problems.

sudo add-apt-repository ppa:glennric/dolphin-emu
sudo add-apt-repository ppa:gregory-hainaut/pcsx2.official.ppa
sudo add-apt-repository ppa:falk-t-j/qtsixa
sudo add-apt-repository ppa:mmbossoni-gmail/emu
sudo add-apt-repository ppa:webupd8team/java
sudo add-apt-repository ppa:jon-severinsson/ffmpeg
#sudo add-apt-repository ppa:mythbuntu/0.27
sudo add-apt-repository ppa:videolan/master-daily
sudo add-apt-repository ppa:stebbins/handbrake-snapshots
sudo add-apt-repository -y ppa:otto-kesselgulasch/gimp
sudo apt-get update
sudo apt-get install qmc2 handbrake-gtk vlc ffmpeg oracle-java7-installer qtsixa pcsx2 dolphin-emu gimp

Things to update from previous installation and security fixes

I always have 2 root partitions with more than 15 GB to install Linux on them. Sometimes I use the second for experiments, but less often than before. My idea was to have Kubuntu on one and Debian (Mepis) on the other, updating Debian to the latest stable and with the latest LTS version of Kubuntu on the other partition. Now that Mepis is not updated (and that different versions of KDE cannot be used simultaneously without many problems) what I do is to install LTS versions of Kubuntu only (clean installations with two years intervals is ok for me), so I installed Kubuntu 14.04 on the other root partition to configure it (and to replace Kubuntu 12.04 in my home desktop too). I always keep the older Linux mounted on /media/OtherLinux.

# Get superuser access (create password for root)
sudo passwd root
# Apache server root directory to /var/www, not /var/www/html (breaks MythWeb)
sudo nano /etc/apache2/sites-available/000-default.conf
# Security with ssh and apache2
sudo cp /media/OtherLinux/etc/hosts.deny /etc/
(add DenyUsers root *@120.114.142.156 to sudo nano /etc/ssh/sshd_config)
(AllowOverride to All and Option Indexes in /var/www with sudo nano /etc/apache2/apache2.conf)
# Mediatomb config
sudo cp /media/OtherLinux/etc/mediatomb/config.xml /etc/mediatomb/
# Reinitialize ssh/apache2
sudo service ssh restart
sudo service apache2 restart
# Check Java version (should be Oracle)
java -version
# Firmware TV tuners
sudo cp /media/OtherLinux/lib/firmware/xc* /lib/firmware/
sudo cp /media/OtherLinux/lib/firmware/dvb_nova* /lib/firmware/
# 1 GB of memory to eclipse (Xmx)
sudo nano /etc/eclipse.ini

In addition to these changes I have a list of .deb packages not available in repositories I always install in my system. Apps like Google Earth 32 bit, Adobe Reader, SkyChart, Skype, and Google Desktop. Google Desktop is an old unsupported app for file indexing, 100 times better than anything available in KDE or Linux in general.

Fixing and fixing and fixing

This is the usual setup after installing the main packages, taking things from my previous installation and fixing some default configuration options in certain packages.

When speaking about bugs or fixes, it is a must to mention the extremely weak grub installer available in the Kubuntu 14.04 CD. It always breaks things (at least when installing grub to the MBR of a disk while having another distribution installed on another disk) so that the system cannot be booted. There is an extremely useful tool you can't live without called boot repair, that can be installed from the live cd or used as a live cd to fix grub. The second option is better, but k3b complains about been unable to properly write a cd with that image… Anyway, it is unacceptable the need of another tool to fix the grub installation made by Kubuntu 14.04, it should be always installed correctly since it is a critical thing. MX-14 does not break things.

Said that, now comes the fixes (besides the vlc and handbrake crashes described above). I will use 'user' to refer to the user login name.

# Don't maximize windows when moving to the top
(Preferences - Workspace behaviour - Screen edges)
# If Akonadi does not work (Have it ever worked after an upgrade?)
rm -r /home/user/.config/akonadi
# If MythWeb does not exists/work
sudo ln -s /usr/share/mythtv/mythweb /var/www/mythweb
(Remove +All and -All in sudo nano /etc/apache2/sites-enabled/mythweb.conf and restart apache)
# Which is my MythTV password ?
sudo more /etc/mythtv/config.xml
# Set swappiness to reduce the use of the swap
sudo sysctl vm.swappiness=20
(Add vm.swappiness = 20 in sudo nano /etc/sysctl.conf to keep it after reboot)
# Less number of Kernel messages
(uncomment 'prevent low level kernel messages' sudo nano /etc/sysctl.conf)
# Key strokes with undesired repeats (typical Dell keyboard hardware problem, also in Windows)
- System settings - Accesibility - Keyboard filter - Bounce keys (Minimum delay here is 100 ms, must be set to 20)
- Edit and /home/user/.kde/share/config/kaccessrc and set BounceKeysDelay=20 instead of 100
- Logout/login. kaccessrc should not be changed, but never go to System settings - Accesibility - Keyboard filter again!
# MythWeb without password by default (in some versions it is created, in other package versions not)
sudo apt-get install apache2-utils
sudo a2enmod auth_digest
htdigest -c /var/www/htdigest MythTV user
sudo chown www-data:www-data /var/www/htdigest 
sudo chmod 640 /var/www/htdigest 
(Uncomment section autentication without any other change except the path of AuthUserFile)
sudo nano /etc/apache2/sites-available/mythweb.conf
sudo service apache2 restart
# MythBackend not running due to lack of time zone data in mysql tables. This happend after some MythTV regular update
- First check the bug is there on the mysql prompt:
mysql -u root -p
> SELECT CONVERT_TZ(NOW(), 'SYSTEM', 'Etc/UTC');
  (it will show null data)
  exit
- Now fix it (replace @ with your root password, no space between -p and password):
mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root -p@ mysql
- Now restart mysql (or reboot. This mandatory step is not mentioned in the MythTV documentation)
sudo service mysql restart
- Check again it is fixed
# Chrome not accepting keyboard input, due to ibus
- shutdown ibus thing (added few days before the final release of Kubuntu 14.04...):
ibus-setup (check to show icon in task bar and close the window, then on the icon right click and 'quit')
- Prevent from using ibus on next boot:
im-config (check primary input is none and automatic is set to ibus. Then accept to change it and set xim as primary input method)
# KMail not showing mails in sent-mail folder although sent process is ok
Check configuration, specially the advanced tab in identities for each account (folders and send method must be ok). Sometimes an update/upgrade changes your KMail configuration...
# Chrome without flash
sudo apt-get install pepperflashplugin-nonfree
# Chrome without Java
Use Firefox, no fix possible in/after Chrome 34. Another option is to install Chrome 31 from Ubuntu raring.
# Some DVDs may be non playable
sudo apt-get install ubuntu-restricted-extras libavcodec-extra libdvdread4
sudo /usr/share/doc/libdvdread4/install-css.sh
# Baloo file indexer taking 100% of CPU, even after adding your
# home to the list of directories that the indexer should skip
(check Indexing-Enabled = false and that the home directory to skip is set as /home/user/ instead of $HOME/)
kate /home/user/.kde/share/config/baloofilerc
(Reboot and check the file did not change. Just in case an emergency shutdown is required, put these commands in a script and call it)
killall -9 baloo_file_cleaner
killall -9 baloo_file
killall -9 baloo_file_extr
(Check also your /home/user/.local/share/baloo/ and remove the xx GB of garbage you may find inside. Don't remove the baloo directory)

And probably I'm forgetting something. Furthermore, I have many useful apps installed but still untested. The conclusion I see after finding most of these bugs in MX14 is that Debian is getting as problematic as Ubuntu. Anyway, at least now I can say my system is stable and running fast without any visible bug.

Good things about Kubuntu 14.04

Of course not everything is that bad about the new release. Akonadi still is and was a pain also in Kubuntu 12.04, as well as Bluetooth support. Bluetooth has improved with 14.04 (no more system freezes with my USB dongle) although it still has minor bugs. In 12.04 I suffered a bug with a 5.1 USB audio card (strong noise coming from the card sometimes), now this is gone in 14.04. I also sent a bug report about a wrong recovering from sleep mode in my laptop with the NVidia card active (with prime also installed), and it was fixed in just a few days.

Few comments about Java and browsers

I'm sad to see the continues steps towards a lack of support of Java applets in browsers. First Oracle forced a warning message about a future lack of support, forcing developers to buy a certificate so that the applet will run … with all priviledges. The explanation is that running applets is dangerous for the users, but running them with all priviledges is something even more dangerous. And now Chrome stops supporting applets.

I have a sky planetarium applet running in my web ephemerides server at www.oan.es/servidorEfem/. It is maybe the biggest applet you have ever seen, requiring the download of more than 40 MB of .jar files. It took many thousands of hours and it was great to provide people with such a tool directly online, but now I see that soon nobody will be able to run it.

2014/05/05 16:39 · Tomás Alonso Albi

Gallery

Selected astronomical images taken with my equipment. As you will see, I'm not a professional photographer, but I do my best. Some of them are old images taken with a film camera, recent ones uses a digital SLR. In addition to the images I have also some videos:

Solar eclipse Solar annular eclipse on October, 3, 2005. It was my first day at OAN so I couldn't use my instrumental. I simply took the camera on my hands.

Venus transit: Venus transit on the Sun on July, 8, 2004. The black drop effect is clearly visible.

Solar eclipse: Solar eclipse of August, 11, 1999, as was shot through my S/C 20 cm telescope.

Log of visits

 
blog.txt · created: 2010/01/31 01:56 (Last modified 2014/10/23 12:51) by Tomás Alonso Albi
 
Recent changes RSS feed Creative Commons License Donate Powered by PHP Valid XHTML 1.0 Valid CSS Driven by DokuWiki